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FOREWORD 


In  the  early  nineties,  the  proposition  that  advanced  distributed  simulation  (ADS)  was  the  wave  of 
the  future  for  test  and  evaluation  (T&E)  was  advanced.  Reaction  was  mixed.  At  one  end  of  the 
spectrum  were  ^people  who  believed  the  need  for  live  testing  would  fade  away,  and  at  the  other 
end  were  people  who  scoffed  at  the  notion  that  ADS  had  any  utility  at  all  for  testers.  At  the 
policy-making  level,  expectations  were  high  and  skepticism  was  subdued.  At  the  implementation 
level,  expectations  were  low  and  skepticism  was  high. 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  program 
was  chartered  in  October  1994  to  conduct  an  objective  assessment  of  the  worth  of  ADS  for 
support  of  T&E.  The  joint  test  force  (JTF)  conducted  tests  in  three  functional  areas:  precision 
guided  munitions  (PGMs);  command,  control,  communication,  computers,  intelligence, 
surveillance  and  reconnaissance  (C4ISR),  and  electronic  warfare  (EW).  While  the  JTF  effort 
was  resource  constrained,  its  results  have  broad  relevance.  Each  of  the  test  areas  is  documented 
in  executive-level  utility  reports.  This  report  addresses  the  utility  of  ADS  in  C41SR  testing. 

This  report  strongly  suggest  that  there  is  very  high  utility  for  ADS  in  C4ISR  testing.  This  utility 
arises  because  1)  C4ISR  systems  operate  in  a  distributed  environment  and  2)  there  is  a 
reasonable  match  between  the  technical  characteristics  of  ADS  and  the  technical  characteristics 
of  command  and  control  architectures.  The  functional  area  of  C4ISR  relies  heavily  on 
information  systems  technology  (1ST),  and  the  turnover  of  that  technology  is  very  rapid  — 
something  on  the  order  of  every  eighteen  months.  If  the  Department  of  Defense  (DoD)  is  to  take 
advantage  of  advances  in  1ST  capabilities  new  information  systems  will  be  continuously  evolving 
requiring  the  application  of  new  and  robust  simulation  tools  for  evaluating  the  suitability  and 
effectiveness  of  these  systems.  Use  of  distributed  simulation  is  particularly  well  suited  for  testing 
in  that  environment. 

This  report  presents  the  results  of  a  C4ISR  Joint  Test  that  implies  the  utility  of  ADS  technology 
as  a  practical  tool  that  can  support  early,  affordable,  and  comprehensive,  testing  of  C4ISR 
systems.  It  has  the  potential  to  enable  robust  system-of-system  testing  as  well  as  mission-level 
and  end-to-end  evaluations.  The  intelligent  application  of  ADS  technology  for  testing  C4ISR 
systems  can  provide  these  benefits  in  a  cost-effective  manner. 

We  believe  that  C4ISR  program  and  test  managers  should  familiarize  themselves  with  ADS  and 
routinely  consider  its  use  in  their  deliberations  and  planning  activities.  It  is  our  hope  that  ADS 
will  be  treated  as  a  readily  available  tool.  While  the  use  of  ADS  will  not  make  sense  in  every 
case,  there  are  many  cases  where  it  not  only  makes  sense,  but  it  may  offer  the  only  practical 
approach  to  realistic  and  rigorous  C4ISR  testing.  We  will  support  and  assist  program  and 
system  managers  who  see  applications  for  ADS  in  their  test  events  and  programs. 

Richard  L.  Lockhart 
Deputy  Director 

Developmental  Test  and  Evaluation 
OUSD(A&T) 


Philip  E.  Coyle 
Director 

Operational  Test  and  Evaluation 


EXECUTIVE  SUMMARY 


1.0  Overview 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  was 
chartered  by  the  Office  of  the  Secretary  of  Defense  in  October  1994  to  investigate  the  utility  of 
advanced  distributed  simulation  (ADS)'  technologies  for  support  of  test  and  evaluation  (T«&E). 
The  JADS  program  is  Air  Force  led  with  Army  and  Navy  participation  and  is  scheduled  to  end  in 
March  2000.  This  report  addresses  the  second  of  three  separate  JADS  tests,  the  End-to-End 
(ETE)  Test,  which  was  completed  in  March  1999. 

The  ETE  Test  evaluated  the  utility  of  ADS  to  support  mission-level  testing  of  command,  control, 
communications,  computers,  intelligence,  surveillance,  and  reconnaissance  (C4ISR)  systems. 
The  test  used  the  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  as  one 
component  of  a  representative  C4ISR  system.  Other  C4ISR  elements  represented  in  the  ETE 
Test  included  virtual  battlefield  entities  (including  about  10,000  threat  entities),  manned  air  and 
ground  operator  workstations,  an  actual  Army  target  analysis  cell,  a  fire  direction  center,  and 
simulated  missiles  for  attacking  selected  targets.  Tactical  communications  systems  were  used 
between  most  of  these  elements.  Figure  ES-1  shows  what  a  typical  C4ISR  ADS  architecture 
might  look  like  using  a  representation  of  the  JADS  ETE  Test  configurations. 


Texas 


Figure  ES-1.  Typical  C4ISR  ADS  Architecture. 


*  ADS  is  a  networking  method  that  permits  the  linking  of  constructive  simulations  (digital  computer  models),  virtual 
simulations  (man-in-the-loop  or  hardware-in-the-loop  simulators),  and  live  players  located  at  distributed  locations 
into  a  single  environment/scenario.  Such  linking  can  result  in  a  more  realistic,  safer,  and/or  more  detailed  evaluation 
of  the  system  under  test. 
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There  were  two  separate  representations  for  the  Joint  STARS  E-8C  aircraft.  In  the  laboratory 
configuration,  the  Virtual  Surveillance  Target  Attack  Radar  System  (VSTARS)  radar  emulation 
represented  the  E-8C  radar  subsystem  and  provided  the  inputs  needed  to  drive  target  displays  on 
operator  workstations.  In  the  live  configuration,  an  actual  E-8C  aircraft  was  flown  and  the 
workstation  operators  observed  actual  live  ground  targets  participating  in  an  exercise  along  with 
the  virtual  battlefield  entities. 

2.0  End-to-End  Test  Results  and  Conclusions 

Within  the  confines  of  the  ETE  Test  data,  our  assessment  is  that  the  architecture  we  employed 
has  utility  for  support  of  C4ISR  T&E,  especially  realistic  mission-level  testing.  The  IADS  data 
indicate  that  developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation 
(OT&E)  activities  incorporating  ADS  technology  are  both  practical  and  cost  effective. 

Key  lessons  learned: 

•  Distributed  testing  often  requires  linkage  among  dissimilar  facilities,  network  equipment,  and 
simulations.  While  practical  and  cost  effective  in  many  cases,  implementation  is  more 
challenging  than  many  people  think.  Plan  for  a  number  of  rehearsals  and  periods  of  “fix” 
time. 

•  ADS  test  requirements  must  be  clearly  defined  early  in  the  test  planning  phase,  since 
individual  facilities  are  generally  unfamiliar  with  conducting  coordinated,  distributed  T&E 
tests. 

•  Instrumentation  and  data  management  are  a  challenge. 

•  Have  a  centralized  test  control  center  with  test  controllers -who  are  extremely  familiar  with 
the  test  and  network  configuration. 

Conclusions: 

•  ADS  testing  of  C4ISR  systems  is  technically  feasible,  provides  valid  data,  and  is  cost 
effective  in  many  cases. 

•  ADS  has  great  potential  as  a  C4ISR  testing  tool  and  provides  a  viable  means  of  conducting 
realistic  mission-level  evaluations. 

3.0  Observations  for  C4ISR  T&E 

Since  all  C4ISR  systems  contain  the  same  basic  elements  (e.g.,  sensor(s),  sensor  platform(s), 
command  and  control  elements,  communication  lines,  and  computer  hardware  and  software),  the 
extension  of  the  ETE  Test  results  to  other  possible  C4ISR  applications  is  relatively 
straightforward.  ADS  technology  allows  the  evaluation  of  human-in-the-loop  actions,  decision 
processes,  timelines,  and  interoperability  which  digital  simulations  do  not  model  well.  Using 
ADS,  a  mission-level  scenario  model  can  be  linked  to  actual  C4ISR  hardware  and  software  with 
tactical  operators-in-the-loop  and  tactical  communications  links  for  realistic  testing  in  force-on- 
force  scenarios  which  cannot  be  accomplished  in  live  testing. 
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A  test  control  center  is  a  requirement  for  all  testing,  distributed  or  not.  Fortunately,  the  EXE  Test 
experience  suggests  that  the  control  center  can  function  from  almost  an5where  with  costs  tailored 
to  the  test  requirements.  Thus,  a  specific  test  may  save  resources  by  using  an  existing  control 
center. 

ADS  is  not  just  of  value  to  C4ISR  T&E  but  can  be  applied  throughout  the  system  acquisition  life 
cycle.  In  fact,  the  benefits  of  using  ADS  are  best  realized  over  the  life  of  a  program.  ADS  is  an 
enabling  technology  for  Simulation  Based  Acquisition  (SBA)  and  the  Simulation,  Test,  and 
,  Evaluation  Process  (STEP)  as  applied  to  C4ISR  systems,  since  these  systems  are  naturally 
distributed  by  nature.  Areas  where  ADS  has  relevance: 

•  ADS  can  create  a  cost-effective  environment  for  test  preparation  to  include  the  development 
of  concepts  of  operations,  refinement  of  test  scenarios,  rehearsal  of  test  execution,  and  data 
collection  and  analysis. 

•  ADS  allows  the  integration  of  models  developed  by  different  acquisition  programs. 

•  ADS  can  expedite  the  association  of  results  from  live  tests  with  output  of  simulations. 

•  ADS  supports  the  execution  of  the  Joint  Vision  2010  paradigm,  which  requires  realistic 
battlespace  environments  populated  with  many  weapon  systems  and  threats.  In  particular, 
ADS  allows  the  large-scale,  complex  environment  evaluations  needed  for  C4ISR  systems. 

•  ADS  can  support  the  model-test-model  process  by  providing  more  realistic  test  results  that 
can  be  used  to  refine  digital  system  models. 

•  ADS  enables  the  linking  and  integration  of  geographically  distributed  resources  from 
different  system  representation  domains  (i.e.,  digital  system  model,  hardware-in-the-loop 
laboratory,  integrated  system  test  facility,  open  air  range)  which  can  lower  testing  costs. 

•  ADS  supports  experimentation  of  emerging  warfighting  concepts  and  testing  new  weapon 
systems. 

•  ADS  can  reduce  the  operations  tempo  (OPTEMPO)  of  test  participants  and  evaluators  in 
pretest  training  and  integration  periods  as  well  as  in  the  test  period  itself. 

The  ETE  Test  results  strongly  suggest  that  ADS  has  excellent  potential  for  improving  C4ISR 
testing  and  system  acquisition.  Thus,  test  plemners  and  program  managers  should  consider  the 
technology  as  a  relevant  tool  for  their  program  unless  an  objective  assessment  suggests 
otherwise. 
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1.0  Purpose  and  Background 

1.1  Report  Purpose 

This  report  summarizes  the  assessment  of  the  utility  of  advanced  distributed  simulation^  (ADS) 
for  the  test  and  evaluation  (T&E)  of  command,  control,  communications,  computers, 
intelligence,  surveillance,  and  reconnaissance  (C4ISR)  systems.  This  assessment  was  based  on 
the  results  and  lessons  learned  from  the  Joint  Advanced  Distributed  Simulation  (JADS)  End-to- 
End  (ETE)  Test  along  with  results  from  other  related  efforts.  The  benefits  of  ADS-based  T&E 
and  training  to  C4ISR  programs  are  also  summarized. 

The  assessment  presented  in  this  report  gives  general  guidelines  for  implementation  of  ADS- 
based  testing  of  C4ISR  systems.  Detailed  instractions  for  linking  specific  C4ISR  systems  using 
specific  architectures  were  not  developed. 

1.2  Joint  Advanced  Distributed  Simulation  Program  Overview 

The  JADS  Joint  Test  and  Evaluation  (JT&E)  was  chartered  by  the  Deputy  Director,  Test, 
Systems  Engineering  and  Evaluation  (Test  and  Evaluation)  ,  Office  of  the  Under  Secretary  of 
Defense  (OSD)  (Acquisition  and  Technology)  in  October  1994  to  investigate  the  utility  of  ADS 
technologies  for  support  of  developmental  test  and  evaluation  (DT&E)  and  operational  test  and 
evaluation  (OT&E).  The  program  is  Air  Force  led  with  Army  and  Navy  participation  and  is 
currently  scheduled  to  end  in  March  2000. 

The  JADS  JT&E  charter  focuses  on  three  issues:  what  is  the  present  utility  of  ADS,  including 
distributed  interactive  simulation  (DIS),  for  T&E;  what  are  the  critical  constraints,  concerns,  and 
methodologies  when  using  ADS  for  T&E;  and  what  are  the  requirements  that  must  be  introduced 
into  ADS  systems  if  they  are  to  support  a  more  complete  T&E  capability  in  the  future. 

The  JADS  JT&E  investigated  ADS  applications  in  three  slices  of  the  T&E  spectrum:  the  System 
Integration  Test  (SIT)  explored  ADS  support  of  air-to-air  missile  testing;  the  ETE  Test 
investigated  ADS  support  for  C4ISR  testing;  and  the  Electronic  Warfare  (EW)  Test  examined 
ADS  support  for  EW  testing.  The  JADS  Joint  Test  Force  (JTF)  was  also  chartered  to  observe  or 
to  participate  at  a  modest  level  in  ADS  activities  sponsored  and  conducted  by  other  agencies  in 
an  effort  to  broaden  conclusions  developed  in  the  three  dedicated  test  areas. 


^  ADS  is  a  networking  method  that  permits  the  linking  of  constructive  simulations  (digital  computer  models),  virtual 
simulations  (man-in-the-loop  or  hardware-in-the-loop  simulators),  and  live  players  located  at  distributed  locations 
into  a  single  environment/scenario.  Such  linking  can  result  in  a  more  realistic,  safer,  and/or  more  detailed  evaluation 
of  the  system  under  test. 

^  This  office  is  now  the  Deputy  Director,  Developmental  Test  and  Evaluation,  Strategic  and  Tactical  Systems. 
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2.0  Supporting  Activities  and  Results 
2.1  End-to-End  Test 


2.1.1  End-to-End  Test  Overview 

The  ETE  Test  evaluated  the  utility  of  ADS  to  support  testing  of  C4ISR  systems  using  the  Joint 
Surveillance  Target  Attack  Radar  System  (Joint  STARS)  as  one  component  of  a  representative 
C4ISR  environment.  The  intent  was  to  provide  a  complete,  robust  set  of  interactions  from  sensor 
to  weapon  system  including  the  additional  intermediate  nodes  that  would  be  found  in  a  tactical 
engagement.  The  test  traced  a  thread  of  the  complete  battlefield  process  from  target  detection  to 
target  assignment  and  engagement  at  corps  level  using  ADS. 

The  ETE  Test  consisted  of  four  phases.  Phase  1  developed  or  modified  the  components  needed 
to  develop  the  ADS  test  environment.  Phase  2  used  a  laboratory  environment  to  evaluate  the 
utility  of  ADS  for  DT&E  and  early  OT&E  applications.  Phase  3  modified  the  actual  Joint 
STARS  equipment  and  verified  its  compatibility  to  interface  with  the  ADS  environment.  Phase 
4  was  a  live,  open  air  test  designed  to  mix  live  and  virtual  targets  and  provide  an  end-to-end 
environment  for  testing  the  Joint  STARS  in  its  operational  environment.  ETE  testing  was 
completed  in  March  1999,  and  results  are  documented  in  a  separate  report  for  each  phase'^  and  in 
various  technical  papers  (see  Appendix  D). 

2.1.2  End-to-End  Test  Approach 

The  test  scenario  was  for  a  Joint  STARS  E-8C  aircraft  to  observe  a  threat  rear  area  of  interest  and 
generate  target  displays  on  workstations  on  board  the  E-8C  and  at  a  light  ground  station  module 
(LGSM)  on  the  ground  via  the  surveillance  and  control  data  link  (SCDL).  The  resulting  radar 
reports  were  provided  to  a  target  analysis  cell  (TAG)  for  analysis  and  target  assignment.  Fire 
support  missions  were  tasked  by  the  TAG  using  the  Advanced  Field  Artillery  Tactical  Data 
System  (AFATDS)  to  a  fire  direction  center  (FDG)  and  resulted  in  the  launch  of  an  Army 
Tactical  Missile  System  (ATAGMS)  missile  against  selected  threat  targets. 

In  the  ETE  Test,  the  threat  ground  entities  (numbering  about  10,000)  were  generated  by  the  Janus 
6.88D  simulation  at  the  U.S.  Army  Training  and  Doctrine  Gommand  Analysis  Genter  (TRAG), 
White  Sands  Missile  Range  (WSMR),  New  Mexico.  An  actual  manned  LGSM  and  TAG  were 
used  at  Fort  Hood,  Texas.  The  TAG  was  linked  to  a  FDG  at  Fort  Sill,  Oklahoma,  by  AFATDS 
terminals  at  Fort  Hood  and  Fort  Sill.  The  ATAGMS  missile  launch,  flyout,  and  detonation  were 


Detailed  infomation  on  Phases  1,  2,  3,  and  4  can  be  found  in  their  respective  reports  available  at 
http;//www.jads.abq.com.  (After  1  March  2000  refer  requests  to  Headquarters  Air  Force  Operational  Test  and 
Evaluation  Center  History  Office  (HQ  AFOTEC/HO),  8500  Gibson  Blvd.  SE,  Kirtland  Air  Force  Base,  New  Mexico 
87117-5558  (phone:  505-846-2579)  or  the  Science  Applications  International  Corporation  (SAIC)  Technical 
Library,  2001  North  Beauregard  St.  Suite  800,  Alexandria,  Virginia  22311  (phone:  703-578-8222).) 
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simulated  by  the  Tactical  Army  Fire  Support  Model  (TAFSM)  simulation  hosted  at  the  Depth 
and  Simultaneous  Attack  Battle  Laboratory  at  Fort  Sill.  Target  engagement  results  were 
determined  by  Janus.  The  Test  Control  and  Analysis  Center  (TCAC)  in  Albuquerque,  New 
Mexico,  provided  test  control,  monitored  the  health  of  the  network,  and  ensured  that  adequate 
data  flowed  in  support  of  the  test.  All  players  were  linked  by  means  of  a  T-1  wide  area  network 
(WAN)  or  by  tactical  communications  links. 

There  were  two  separate  representations  for  the  E-8C  aircraft. 

•  In  the  laboratory  configuration  tested  in  Phase  2  (Fig.  1),  the  Virtual  Surveillance  Target 
Attack  Radar  System  (VSTARS)  emulation  represented  the  E-8C  radar  subsystem  and 
provided  the  inputs  needed  to  drive  target  displays  on  Advanced  Technology  Work  Stations 
(ATWS)  (representing  displays  on  board  the  E-8C  aircraft)  and  the  LGSM.  VSTARS  was 
operated  at  the  Northrop  Grumman  Surveillance  and  Battle  Management  Systems  facility  in 
Melbourne,  Florida.  SCDL  traffic  was  transmitted  over  a  separate,  dedicated  T-1  link 
between  Melbourne  and  Fort  Hood. 

•  In  the  live  configuration  tested  in  Phase  4  (Fig.  2),  an  actual  E-8C  aircraft  was  flown  over 
Fort  Hood  and  observed  live  targets  participating  in  an  exercise.  Entity  state  data  for  the 
virtual  entities  simulated  by  Janus  were  uplinked  to  the  aircraft  via  a  satellite 
communications  (SATCOM)  link  (the  uplink  feed  was  from  the  Northrop  Grumman  facility 
in  Melbourne,  Florida),  and  simulated  radar  returns  for  the  virtual  entities  were  inserted  into 
the  radar  stream.  The  radar  signals  for  both  the  live  and  the  virtual  targets  were  processed 
together  on  board  the  aircraft  resulting  in  nearly  seamless  displays  on  the  ATWS. 


LOS  =  line-of-sight 

Figure  1.  ETE  Test  Laboratory  Configuration 
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Figure  2.  EXE  Test  Live  Connguration 


Entity  state  and  ATACMS  fire  and  detonate  messages  were  transmitted  among  the  players  using 

DIS  protocol  data  units.  However,  the  C4ISR-related  communications  employed  tactical 

protocols  and  doctrinally  correct  means: 

•  Tactical  SCDL  messages  were  used. 

•  The  LGSM  communicated  with  the  TAG  using  the  Common  Ground  Station-100  (CGS-100), 
a  subsystem  of  the  Compartmented  All  Source  Analysis  System  (ASAS)  Message  Processing 
System  (CAMPS). 

•  The  TAC  communicated  with  the  FDC  using  tactical  AFATDS  messages. 

2.1.3  End-to-End  Test  Results 

The  key  results  from  ETE  testing  (see  ETE  Test  reports  for  details): 

•  Both  ADS  configurations  generated  valid  Joint  STARS  data,  as  judged  by  subject  matter 
experts. 

•  It  was  specifically  validated  that  Janus  6.88D  represented  vehicle  behavior  to  the  degree 
detectable  by  the  Joint  STARS,  as  judged  by  Joint  STARS  operators  viewing  vehicle 
movement  presented  by  the  Joint  STARS  operator  workstation. 

•  Test  participants  could  not  identify  any  Joint  STARS  radar  simulation  process  or  function 
that  limited  their  ability  to  perform  their  mission/job  or  altered  their  approach  to  their 
mission/job. 

•  The  credibility  of  the  Joint  STARS  data  was  enhanced  by  the  use  of  actual  operational 
hardware  (e.g.,  ATWS,  LGSM,  AFATDS)  with  trained  operators  whenever  possible. 
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•  Sufficient  data  were  collected  to  support  evaluation  of  most  of  the  Joint  STARS  multiservice 
operational  test  and  evaluation  measures.  These  measures  were  not  evaluated  by  IADS 
analysts  since  the  IADS  objective  was  to  evaluate  ADS-based  testing  not  Joint  STARS. 

•  When  live  and  virtual  entities  were  mixed  using  the  live  ETE  Test  configuration,  it  was 
impossible  to  distinguish  between  them  based  upon  size,  color,  contrast,  and  movement. 
However,  the  live  and  virtual  areas  could  be  distinguished  based  on  the  different  scenarios 
used  and  civilian  traffic  “clutter.”  The  live  area  contained  civilian  traffic  around  Fort  Hood, 
and  the  virtual  area  contained  simulated  battlefield  entities  in  Iraq.  If  the  live  entities  had 
simulated  a  battle  in  Iraq,  the  live  and  virtual  entities  would  have  been  indistinguishable. 

•  The  ETE  Test  required  only  a  small  fraction  of  the  T-1  bandwidth  (1.544  megabits  per 
second).  This  indicates  that  a  much  higher  data  transfer  rate  (e.g.,  more  nodes  or  entities) 
could  be  applied  to  ADS  testing  without  causing  bandwidth  problems. 

•  The  live  testing  using  the  SATCOM  link  to  the  E-8C  aircraft  showed  that  all  the  available 
SATCOM  link  bandwidth  was  required  for  data  transmission,  and  that  buffering  was  needed 
at  times  to  handle  periods  of  heavy  scenario  activity.  Without  buffering,  the  SATCOM  link 
exhibited  an  average  latency  of  around  two  seconds.  With  buffering,  the  latency  approached 
six  seconds. 

•  The  ETE  Test  exhibited  acceptable  latency  values.  In  other  words,  the  latencies  experienced 
during  testing  did  not  affect  the  validity  of  test  results. 

•  The  ETE  Test  network  was  highly  reliable  during  testing,  largely  because  of  the  ETE  Test 
team’s  extensive  pretest  risk  reduction  efforts. 

•  Test  control  procedures  were  refined  throughout  the  preparation  process  and  worked  well 
during  testing. 

2.1.4  End-to-End  Test  Lessons  Learned 

The  key  technical  lessons  learned; 

•  Simulations,  when  used  in  ADS  testing,  should  be  carefully  planned  and  developed.  Using 
reliable  simulations  is  important  for  a  successful  test. 

•  Distributed  testing  often  requires  linkage  among  dissimilar  facilities,  network  equipment,  and 
simulations.  However,  careful  planning  can  significantly  reduce  the  potential  for  difficulties 
arising  from  network  interface  problems. 

•  Time  sources  must  be  synchronized  using  a  “master  clock”  and  then  validated  at  each 
network  node. 

•  Special  test  equipment  and  networking  tools  are  necessary  for  distributed  testing.  The  tool 
set  must  be  able  to  rapidly  isolate  the  specific  cause  of  network  and  ADS/DIS  problems. 

The  key  infrastructure  and  process  lessons  learned: 

•  ADS  test  requirements  must  be  clearly  defined  early  in  the  test  planning  phase,  since 
individual  facilities  are  generally  unfamiliar  with  conducting  coordinated,  distributed  tests. 

•  Develop  contingency  plans  that  allow  quick  decisions  on  alternatives  and  prevent  unexpected 
equipment  and  communications  problems  from  completely  disrupting  the  test. 


•  Use  a  stepped  approach  to  testing  where  each  successive  ADS  test  builds  on  the  success  of 
earlier  tests.  This  “test,  analyze,  fix,  test”  approach,  in  concert  with  structured,  independent 
testing  of  the  network,  will  greatly  improve  the  chances  for  successful  ADS  testing. 

•  Risk  reduction  testing  prior  to  actual  test  execution  provided  effective  rehearsals  and  was 
helpful  for  troubleshooting. 

•  Detailed  planning  for  data  management  is  necessary  before  testing  commences. 

•  Have  a  centralized  test  control  center  with  test  controllers  who  are  extremely  familiar  with 
the  test  and  network  configuration. 

•  Personnel  involved  in  a  distributed  test  need  to  understand  the  “big  picture.”  When  people 
are  geographically  separated,  it  becomes  easy  for  them  to  focus  on  their  own  individual 
portion  of  the  test.  When  problems  arise,  personnel  who  understand  the  entire  test  and  the 
overall  network  will  find  solutions  much  faster. 

2.1.5  End-to-End  Test  Conclusions 

The  ETE  Test  live  ADS  configuration  has  utility  for  realistic  OT&E  to  include  mission-level 

testing  of  C4ISR  systems. 

•  In  contrast  to  ADS-supported  testing,  most  traditional  live  C4ISR  testing  cannot  be  done  at 
the  mission  level  because  of  the  unavailability  and  prohibitive  cost  of  involving  the  required 
numbers  of  live  battlefield  entities  and  limited  access  to  key  real  world  systems  because  of 
their  limited  numbers  and  high  usage. 

•  The  use  of  ADS  permits  the  number  of  targets  to  be  greatly  augmented  by  the  seamless 
integration  of  live  and  virtual  threat  entities  while  operating  the  sensor/sensor  platform  in  the 
actual  operational  environment. 

•  Realism  and  the  credibility  of  test  results  are  enhanced  by  the  use  of  actual  operational 
hardware  by  trained  operators  and  by  the  employment  of  the  sensor  in  a  real  open-air 
environment  subjecting  it  to  actual  background  and  clutter  effects. 

The  ETE  Test  laboratory  ADS  configuration  has  utility  for: 

•  C4ISR  DT&E,  since  laboratory/brass  board  system  under  test  configurations  can  be  used,  and 
multiple  repetitions  of  the  same  scenario  can  be  performed  for  parametric  testing. 

•  Realistic  rehearsal  and  refinement  of  live  test  scenarios,  since  multiple  repetitions  of  the 
scenarios  can  be  performed  using  the  same  equipment  operators  as  the  live  test. 

•  Early  operational  assessment  of  C4ISR  system  components  prior  to  building  them  by  using 
laboratory/brass  board  configurations  for  the  new  components  (e.g.,  sensor/sensor  platforms) 
linked  to  existing  tactical  hardware  manned  with  actual  operators  for  the  other  components 
(e.g.,  command  and  control  [C2]  elements).^ 

Both  ADS  configurations  have  utility  for  C4ISR  system  training. 

•  Conventional  training  can  be  limited  by  the  availability  of  those  assets  making  up  the  C4ISR 
system’s  operational  environment.  ADS  technology,  by  simulating  those  key  assets,  can 
provide  longer  periods  of  time  for  realistic  operation  of  the  C4ISR  system. 


^  The  assumption  here  is  that  only  one  component/element  of  an  existing  C4ISR  system  is  being  upgraded,  and  the 
other  elements  are  to  be  unchanged. 
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•  C4ISR  system  operators  can  take  advantage  of  the  additional  training  time  provided  by  ADS 
technology  to  confirm  current  tactics  and  to  test  “what  if’  scenarios  and  new  tactics. 

•  ADS  simulations  can  help  C4ISR  system  operators  familiarize  themselves  with  the  maneuver 
tactics  of  foreign  armed  forces  -  valuable  experience  for  possible  future  deployments. 

2.2  Supporting  Studies  Summary 

In  addition  to  the  ETE  Test  results,  other  studies  were  performed  both  inside  and  outside  of  the 

JADS  JTF.  The  findings  of  those  studies  relevant  to  C4ISR  ADS  applications  are  summarized  as 

follows: 

•  DIS-compatible  interfaces  are  being  developed  that  will  permit  combat  scenario  simulations 
to  provide  the  battle  stimuli  for  the  operational  testing  of  C4I  systems  [Ref.  1]. 

•  DIS-based  programs  have  been  proposed  for  testing  command,  control,  and  communications 
(C3)  [Ref.  2],  modeling  airborne  reconnaissance  systems  [Ref.  3],  prototyping  twenty-first 
century  command  and  control  environments  [Ref.  4],  simulating  C2  [Ref.  5],  C3  [Ref.  6],  and 
command,  control,  communications  and  intelligence  (C3I)  environments  [Ref.  7]. 

•  Distributed  Theater  Missile  Defense  (TMD)  testing  is  being  performed  using  the  Theater 
Missile  Defense  System  Exerciser  (TMDSE)  at  the  Joint  National  Test  Facility  (JNTF). 
Using  DIS  protocol  data  units  (PDUs),  TMDSE  injects  a  real-time,  common  threat  scenario 
into  real,  geographically  distributed  tactical  sensors  and  weapon  systems.  The  tactical 
systems  respond  in  real  time  via  their  respective  tactical  communication  data  nets  allowing 
each  individual  TMD  system  to  operate  synergistically  in  a  tactically  realistic  battlefield. 
TMDSE  provides  a  real-time,  hardware-in-the-loop,  software-in-the-loop,  tactical 
communications-in-the-loop,  tactical  operator-in-the-loop,  geographically  distributed 
interactive  test  capability  [Ref  8]. 

•  The  Joint  Theater  Missile  Defense  (JTMD)  Attack  Operations  (AO)  JT&E  used  a  large-scale, 
man-in-the-loop  distributed  simulation  architecture  to  investigate  the  capability  of  a 
commander  in  chief  to  integrate  near-term  reconnaissance  and  surveillance  sensors,  C4I 
assets,  and  attack  systems  in  order  to  conduct  TMD  attack  operations  in  several  theaters  of 
operations  with  varying  force  structures  [Refs.  9, 10]. 

•  The  Simulation  Interoperability  Standards  Organization  (SISO)  Implementation  Study  Group 
for  C4I-to-Simulation  Interfaces  (ISG-C4I)  has  been  formed  to  aid  in  the  standardization  of 
the  interfacing  between  real  C4I  and  simulations  [Ref.  11]. 

•  The  growing  importance  of  the  high  level  architecture  (HLA)  is  reflected  in  discussions  on 
the  implementation  of  a  prototype  C4I  federation  object  model  [Ref.  12]  and  the  development 
of  a  C4ISR  analysis  federation  [Ref.  13]. 
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•  The  use  of  HLA^  overcomes  many  limitations  and  undesirable  features  of  DIS  standards  by 
allowing  data  protocol  flexibility,  eliminating  unnecessary  coordinate  transforms,  and 
reducing  the  amount  of  data  transmitted  between  nodes  by  only  transmitting  entity  attributes 
that  have  changed  [Ref.  14].  HLA  implementation  for  C4ISR  scenarios  has  already  been 
demonstrated  in  several  programs. 

•  The  Synthetic  Theater  of  War  (STOW)  team  has  demonstrated  its  entity-level  simulation 
in  the  first  large-scale,  (3,700  platforms  and  8,000  objects)  HLA-compliant  exercise 
utilizing  the  runtime  inffastmcture  (RTI)  [Ref.  15].  The  STOW  exercise  included  a  Joint 
STARS  federate  [Ref.  16]. 

•  The  Simulation,  Testing,  Operations,  Rehearsal  Model  (STORM)  is  being  developed  as 
an  HLA-compliant  distributed  testing  tool  for  the  Army’s  Force  XXI  Battle  Command 
Brigade  and  Below  (FBCB2)  [Ref.  17].  STORM  will  permit  transmission  of  simulation 
information  to  and  from  live  forces  allowing  for  a  relatively  “seamless”  synthetic 
battlefield  environment. 

•  The  Joint  Simulation  System  (JSIMS)  program  is  developing  HLA-compliant  interfaces 
among  simulations  and  operational  C4I  systems  [Ref.  18]. 

•  Project  Constellation  is  a  Test  and  Evaluation  Command  (TECOM)  Virtual  Proving  Ground 
(VPG)  initiative  to  integrate  testing  and  simulation/stimulation  tools  for  distributed  testing  of 
systems.  Project  Constellation  has  used  distributed  sensor-to-shooter  test  beds  with  C4ISR 
elements  to  identify  and  solve  distributed  testing  issues,  especially  those  related  to  controlling 
the  distributed  environment  [Ref.  19]. 

•  The  Foundation  Initiative  2010  (FI  2010)  project,  sponsored  by  the  Central  Test  and 
Evaluation  Investment  Program  (CTEIP),  is  designed  to  provide  test  capabilities  to  upgrade 
the  aging  Department  of  Defense  (DoD)  ranges,  facilities,  and  simulations  [Ref.  20].  FI  2010 
products  will  provide  cost-effective  interoperability  among  test  resources,  across  phases  of 
the  acquisition  process,  and  between  test  resources  and  C4ISR  resources.  These  products 
will  support  linked  tests  and  exercises  that  integrate  resources  from  the  live,  virtual,  and 
constmctive  domains  across  all  mission  areas,  services,  and  all  test  resource  categories. 

•  The  FI  2010  project  will  also  provide  a  common  range  architecture  called  the  Test  and 
Training  Enabling  Network  Architecture  (TENA)  to  support  (a)  test  and  training  range 
operations,  (b)  development  of  range  instrumentation,  (c)  easy  assembly  and  configuration  of 
test  resources,  (d)  execution  of  test  events,  and  (e)  analysis  and  reporting  of  test  event  results 
[Ref.  20]. 


*  HLA  has  been  mandated  by  the  Department  of  Defense  (DoD)  for  linking  distributed  simulations  [Ref.  24].  HLA 
uses  a  standard  set  of  rules,  tools,  and  a  runtime  infrastructure  (RTI)  to  allow  simulations  to  send  and  receive 
information.  HLA  allows  the  simulations  to  treat  communications  in  a  more  abstract  manner  by  providing  a  coimnon 
interface  specification  and  isolating  the  simulations  from  communications  protocol  implementations.  Because 
simulations  can  be  less  aware  of  each  other,  HLA  should  encourage  reuse  of  existing  models  for  different 
applications.  There  are  also  no  standard  data  formats  analogous  to  DIS  PDUs.  Users  can  form  their  own  messages 
based  on  their  needs.  Likewise,  users  can  use  PDUs  if  they  feel  that  the  PDUs  represent  the  data  necessary  for  their 
simulation.  In  most  cases,  the  conclusions  from  DIS-based  testing  should  directly  apply  to  HLA-based  distributed 
testing. 
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3.0  Overall  Advanced  Distributed  Simulation  Utility  Assessment 

3.1  JADS  Issues 

The  JADS  JT&E  program  was  chartered  to  investigate  the  utility  of  ADS  for  both  DT&E  and 
OT&E.  The  charter  letter  identifies  three  issues  to  be  addressed  [Ref.  21]. 

•  Investigate  the  present  utility  of  ADS,  including  DIS,  for  T&E.  The  utility  assessment 
includes  evaluating  the  validity  of  data  from  tests  using  ADS  and  the  benefits  of  using  ADS 
in  T&E. 

•  Identify  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E. 

•  Identify  the  requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to  support  a 
more  complete  T&E  capability  in  the  future. 

The  ability  of  ADS  to  support  C4ISR  T&E  will  be  assessed  in  terms  of  these  issues. 

3.2  General  Utility  of  ADS  for  T&E  of  C4ISR  Systems 

The  assessments  in  this  section  represent  extrapolations  of  the  results  of  JADS  testing  and  related 
ADS  programs  and  are  based  on  JADS  experience  and  insight  from  executing  the  JT&E  without 
rigorous  analysis  or  supporting  data.  Rationales  for  the  assessments  are  given,  as  appropriate. 

3.2.1  General  Utility  Assessment 

ADS  has  utility  for  C4ISR  DT&E  and  OT&E. 

•  As  demonstrated  by  the  success  of  the  JADS  ETE  Test,  ADS-supported  tests  can  provide 
valid  mission-level  C4ISR  performance  evaluations  in  a  number  of  areas  addressed  by  both 
DT&E  and  OT&E. 

•  ADS-supported  tests  can  mitigate  or  eliminate  traditional  T&E  shortfalls  especially  for  force- 
on-force,  system-of-systems  scenarios  and  permit  following  the  thread  of  a  complete 
battlefield  process  (e.g.,  sensor-to-shooter  applications).  In  fact,  ADS-supported  tests  may  be 
the  only  way  to  generate  the  data  needed  to  calculate  some  effectiveness  and  performance 
measures.  Traditional  C4ISR  testing-related  shortfalls  that  ADS  can  overcome  include  [Ref. 
22]; 

•  human  interaction  not  represented, 

•  nonrepresentative  force  levels, 

•  inadequate  quantity  and  types  of  threat  systems,  targets,  and  friendly  systems, 

•  insufficient  test  articles, 

•  unrealistic  test  scenarios,  and 

•  insufficient  number  of  test  events. 

•  The  JADS  JTF  has  judged  ADS  to  have  significant  utility  in  all  mission  areas  evaluated 
(C4ISR,  precision  guided  munitions,  and  EW).  However,  the  strongest  utility  was  found  to 
be  for  testing  C4ISR  systems  or  those  systems  that  interact  with  C4ISR  systems. 
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•  There  are  significant  benefits  in  using  ADS  for  C4ISR  T&E  (see  Section  3.2.2). 

3.2.2  General  ADS  Benefits 

The  benefits  of  ADS-supported  testing  are  best  realized  when  this  testing  approach  is  added  to  a 
total  C4ISR  testing  program.  ADS-supported  tests  are  not  meant  to  replace  any  of  the  current 
testing  techniques  (e.g.,  analytical  studies,  laboratory/field  tests,  hardware-in-the-loop  tests, 
installed  system  test  facility  tests,  live  tests),  but  rather  to  supplement  current  techniques  and 
provide  a  more  comprehensive  evaluation  of  a  C4ISR  system.  When  the  appropriate  mix  of 
testing  techniques  is  used,  benefits  are  realized  from  the  addition  of  ADS-supported  testing. 

The  primary  benefits  of  ADS-supported  C4ISR  testing: 

•  Distributed  test  linking  techniques  permit  realistic  system-of-systems,  mission-level 
evaluations  to  be  performed  at  affordable  costs.  The  realism  results  from  the  combination  of 

•  large  numbers  of  virtual  battlefield  entities  augmenting  a  limited  number  of  live  entities, 

•  large  virtual  battlespace  with  an  operationally  realistic  scenario  driver  and  accredited 
scenario, 

•  human  operators  and  decision-makers-in-the-loop  using  actual  workstations, 

•  communicating  by  the  use  of  tactical  message  protocols,  and 

•  real  sensors  and  sensor  platforms  operating  in  the  actual  environment. 

•  An  ADS-supported  test  allows  the  evaluation  of  the  Value  of  a  C4ISR  system  to  the  overall 
mission  accomplishment.  The  same  scenario  can  be  replayed  with  and  without  the  C4ISR 
system  to  determine  the  impact  of  the  system  on  the  outcome  of  the  battle. 

Other  benefits  include  the  following. 

•  ADS  provides  an  efficient  method  for  C4ISR  interoperability  testing.  Instead  of  having  to 
take  a  particular  system  to  separate  test  facilities  for  each  element  with  which  it  must 
interoperate  in  a  sequential  fashion,  all  facilities/elements  can  be  linked  together  for 
concurrent  testing,  saving  time  and  money.  Concurrent  linked  testing  also  simplifies  the 
analysis  and  correlation  of  test  results  since  a  common  scenario  and  testing  environment  is 
achieved. 

•  The  use  of  ADS  in  support  of  testing  provides  greater  scalability  and  flexibility  than  live 
testing.  The  number  of  simulated  entities  can  be  readily  scaled  for  a  particular  scenario  by 
the  use  of  a  digital  model.  Also,  the  type  of  representation  for  a  given  player  (i.e.,  live, 
virtual,  or  constmctive)  can  be  changed  based  on  considerations  of  cost,  schedule,  fidelity, 
operational  availability,  etc. 

•  Dedicated  ADS-supported  tests  have  advantages  over  piggybacking  on  training  exercises. 

•  Scheduling  conflicts  with  the  training  exercise  are  eliminated. 

•  Dedicated  ADS-supported  tests  are  more  controllable,  since  there  is  no  conflict  over 
testing  and  training  objectives  and  since  virtual  entities  can  be  precisely  controlled. 

•  Virtual  entities  do  not  need  instrumentation  and  are  not  subject  to  instmmentation  errors. 

•  ADS-supported  tests  can  be  repeated  with  the  same  scenario  and  synthetic  environment  each 
time  for  greater  confidence  in  results. 
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•  ADS-supported  tests  can  be  used  to  verify  software  and  techniques  for  sensor  fusion. 
Multiple  sensor  representations  can  be  linked  with  a  common  scenario/synthetic  environment 
and  sensor  outputs  can  be  fed  into  the  actual  fusion  software/techniques. 

•  The  use  of  simulations  for  sensor  platforms  and  battlefield  entities  means  that  any 
conceivable  test  case  or  scenario  can  be  run  without  safety  or  asset  availability  concerns. 

•  Live  tests  can  be  realistically  rehearsed  using  ADS  (since  the  actual  scenarios  and  equipment 
operators  can  be  used),  resulting  in  more  productive  live  testing. 

•  The  same  ADS  configuration  can  be  used  for  DT  and  OT  testing,  as  well  as  realistic  training. 
Only  the  type  and  quantity  of  instrumentation  need  to  be  changed  as  required  by  the  DT,  OT 
or  training  objectives. 

•  ADS  can  reduce  the  operations  tempo  OPTEMPO  of  test  participants  and  evaluators  during 
pretest  training  and  integration  as  well  as  during  the  test  itself.  Distributed  testing  allows 
personnel  to  participate  from  their  home  locations  eliminating  travel  time,  and  realistic 
training  and  rehearsal  periods  result  in  more  efficient  test  preparations. 

3.2.3  General  Role  of  ADS  in  C4ISR  System  Acquisition  Life  Cycle 

The  Department  of  Defense  is  seeking  to  streamline  the  acquisition  process  by  the  use  of 
simulation  technology  through  a  strategy  called  Simulation  Based  Acquisition  (SBA).  The  goals 
of  SBA  are  to  [Ref.  23] 

•  substantially  reduce  the  time,  resources,  and  risk  associated  with  the  entire  acquisition 
process, 

•  increase  the  quality,  military  worth,  and  supportability  of  fielded  systems  while  reducing  total 
ownership  costs  throughout  the  total  life  cycle,  and 

•  enable  integrated  product  and  process  development  across  the  entire  acquisition  life  cycle. 

A  key  element  of  SBA  is  the  Simulation,  Test,  and  Evaluate  Process  (STEP),  which  is  defined  as 
“an  iterative  process  that  integrates  simulation  and  test  for  the  purpose  of  interactively  evaluating 
and  improving  the  design,  performance,  joint  military  worth,  survivability,  suitability,  and 
effectiveness  of  systems  to  be  acquired  and  improving  how  those  systems  are  used  [Ref.  24].” 

Under  the  SBA  and  STEP  concepts,  the  cost  of  testing  is  reduced  because  investments  in 
simulations  in  early  acquisition  phases  can  be  leveraged  rather  than  duplicated  in  later  stages. 
Reliability  is  increased  because  of  the  iterative  approach  inherent  to  SBA  and  STEP  where 
results  from  field  tests  are  incorporated  back  into  models  in  a  “model-test-model”  paradigm. 
Overall  cost  of  acquisition  is  reduced  because  system  evaluators  can  merge  information  from 
multiple  acquisition  phases  maximizing  insight  to  system  performance. 

ADS  is  particularly  useful  in  supporting  SBA  and  STEP  as  applied  to  C4ISR  systems,  since  these 
systems  are  distributed  by  nature.  Areas  where  ADS  has  relevance’: 

•  ADS  can  provide  a  framework  for  the  integration  of  models  developed  by  different 
acquisition  programs,  since  the  use  of  ADS  techniques  (e.g.,  linking  via  network  interfaces 


’  As  in  Sections  3.2.1  and  3.2.2,  the  assessments  in  this  section  are  extrapolations  of  the  results  of  JADS  testing  and 
related  ADS  programs  and  are  based  on  JADS  experience  and  insight. 
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and  data  protocols)  can  permit  disparate  simulations  to  be  linked,  as  demonstrated  during 
JADS  testing. 

•  ADS  can  expedite  the  association  of  results  from  live  tests  of  a  C4ISR  element  with  the 
output  of  simulations  of  the  element.  Using  the  same  approach  as  the  EXE  Test,  the  element 
can  either  be  represented  by  the  actual  hardware  or  by  a  simulation  and  is  linked  in  either 
case  to  the  same  representations  of  the  other  elements  of  the  C4ISR  system.  Since  the  same 
scenarios  and  synthetic  environment  can  be  used,  correlation  of  results  between  the  live 
element  test  and  the  element  simulation  is  relatively  straightforward  and  can  support  the 
model-test-model  process  at  the  element  level. 

•  ADS  can  also  support  the  model-test-model  process  at  the  system-of-systems  level  by 
providing  more  realistic  mission-level  test  results  that  can  be  used  to  refine  digital  system 
models  for  the  entire  C4ISR  system. 

•  ADS  supports  the  execution  of  the  Joint  Vision  2010  paradigm  which  requires  realistic 
battlespace  environments  populated  with  many  weapon  systems  and  threats.  In  particular, 
ADS  allows  the  large-scale,  complex  environment  evaluations  needed  for  C4ISR  systems. 

•  ADS  enables  the  linking  and  integration  of  geographically  distributed  resources  from 
different  system  representation  domains  (i.e.,  digital  system  model  (DSM),  hardware-in-the- 
loop  lab,  integrated  system  test  facility,  battle  labs,  open  air  range)  which  can  lower  test  costs. 

•  ADS  supports  experimentation  of  emerging  warfighting  concepts  and  testing  new  weapon 
systems. 

Further,  ADS  techniques  can  be  applied  to  all  C4ISR  element  acquisition  phases. 

•  Concept  Exploration.  If  a  C4ISR  element  DSM  becomes  available  during  this  initial  system 
acquisition  phase,  ADS  linking  techniques  can  be  used  to  provide  a  more  realistic  battlefield 
environment  and  to  permit  human  interactions  with  the  simulated  element.  This  capability  is 
especially  useful  for  development  of  a  concept  of  operation  (ConOps)  for  the  emerging 
element  system. 

•  Program  Definition  and  Risk  Reduction.  As  prototype  C4ISR  element  system  hardware  is 
developed,  ADS-based  testing  can  be  expanded  and  refined.  ADS  configurations  can  be  used 
to  support  early  operational  assessments  and  for  more  realistic  specification  compliance 
testing  during  DT&E. 

•  Engineering  and  Manufacturing  Development.  Mission-level  evaluations  have  replaced 
traditional  requirements-based  OT&E  to  determine  whether  the  system  supports  the 
warfighter  [Ref.  25],  and  ADS-based  testing  is  well  suited  for  this  application.  By  using 
ADS,  the  C4ISR  system  can  be  placed  in  a  realistic  operational  environment  and  valid  data 
can  be  collected  for  the  evaluation  of  operational  measures. 

•  Production,  Fielding/Deployment  and  Operational  Support.  By  permitting  operator-in-the- 
loop  operations  with  tactical  hardware,  ADS  can  support  the  development  of  tactics  and 
operational  procedures  in  conjunction  with  realistic  training.  ADS  can  also  be  used  for  the 
development  of  integrated  logistics  concepts. 

ADS  is  also  useful  for  supporting  the  iterative  spiral  development  process.  As  the  C4ISR 

element  undergoes  improvements  and  upgrades,  ADS  can  be  used  to  more  realistically  to 

•  verify  the  element  system’s  design, 

•  confirm  that  design  risks  have  been  controlled. 
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•  certify  readiness  for  operational  testing,  and 

•  evaluate  the  system’s  operational  effectiveness,  suitability,  and  survivability. 

3.3  Critical  Constraints,  Concerns,  and  Methodologies  for  C4ISR  T&E 

Constraints  with  respect  to  the  application  of  ADS  technology  can  be  summarized  as  follows. 

•  There  are  limited  tools  (e.g.,  network  design  software,  network  test  and  analysis 
hardware/software,  distributed  test  control  hardware/software),  expertise,  and  resources  for 
implementing  ADS-based  testing. 

•  There  are  fundamental  ADS-related  constraints. 

•  Latency  is  not  normally  a  limitation  for  C4ISR  applications. 

•  Bandwidth  can  limit  the  number  of  entities  represented  in  a  scenario  or  their  update  rate. 
However,  this  was  not  an  issue  for  the  ETE  Test  involving  about  10,000  entities. 

•  Bandwidth  limitations  can  also  make  it  difficult  to  pass  detailed  environmental  data 
among  players.  The  standard  practice  is  to  employ  a  common  environmental  databeise  at 
each  node  to  ensure  that  all  simulated  sensors  see  the  same  virtual  environment. 

•  Special  interfaces  are  needed  to  inject  virtual  entities  into  live  systems.  Such  interfaces 
may  be  used  to  inject  virtual  entities  behind  a  live  sensor  but  before  sensor  signal 
processing,  as  was  done  in  the  ETE  Test. 

There  are  concerns  which  must  be  addressed  for  the  proper  implementation  of  ADS.  These 
include  the  following: 

•  ADS  implementation  adds  another  layer  requiring  management  oversight.  This  is  minimized 
by  integrating  the  ADS  implementers  into  the  test  force,  as  demonstrated  during  the  JADS 
JT&E. 

•  ADS  implementation  requires  additional  infrastructure  in  a  test,  and  currently  there  is  little  or 
no  existing  infrastructure.  However,  there  are  DoD-sponsored  options  for  the  WAN  portion 
of  the  infrastructure  including  the  Defense  Research  and  Engineering  Network  (DREN), 
Defense  Simulation  Internet  (DSI),  and  Secure  Internet  Protocol  Router  Network 
(SIPRNET). 

•  The  integration  of  various  data  collected  at  distributed  sites  into  a  centralized  data  set  can  be 
difficult.  However,  proper  planning  can  greatly  mitigate  this.  In  fact,  the  WAN  can  be  used 
post-test  to  transmit  data  collected  at  the  distributed  sites  to  a  central  analysis  facility,  as  done 
during  JADS  testing. 

.  •  Potential  implementers  don’t  know  where  to  go  to  get  information  and  help  on  implementing 
ADS,  e.g.,  available  simulations,  facilities,  and  tools.  This  concern  is  addressed  in 
appendices  to  this  report. 

•  Appendix  C  -  checklist  for  implementing  ADS-based  tests 

•  Appendix  D  -  bibliography  of  all  JADS  ETE-related  reports  and  papers 

•  Appendix  E  -  list  of  ADS-related  Web  sites 

•  The  cost  of  implementing  ADS  is  a  common  concern.  There  can  be  significant  up-front 
costs,  since  most  existing  facilities  were  not  designed  to  be  linked  and  will  thus  require  some 
modification  along  with  the  appropriate  interfaces  before  linking  is  possible.  However,  this 
cost  can  be  amortized  over  all  phases  of  a  C4ISR  development  and  testing  program  and  even 
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across  related  C4ISR  programs.  Also,  it  should  be  noted  that  the  cost  per  byte  of  data  is 
generally  less  than  for  a  live  test,  since  a  distributed  test  can  generate  considerably  more  data 
than  a  live  test. 

•  Distributed  people  and  resources  require  effective  communications  and  frequent 
coordination,  but  this  can  be  done  with  the  right  organization  and  lines  of  communications. 

•  Security  can  be  a  concern  when  linking  facilities  with  different  security  requirements. 
Although  special  attention  must  be  given  to  developing  and  coordinating  security  procedures, 
the  process  is  straightforward.  For  example,  the  ETE  Test  linked  classified  and  unclassified 
facilities. 

•  Concerns  over  the  availability  of  specific  ranges  and  facilities  can  be  alleviated  by  identifying 
substitute  resources. 

•  Effective  centralized  control  of  distributed  tests  can  be  challenging,  but  it  can  be  done  with 
proper  planning  and  tailored  investment. 

A  number  of  methodologies  apply  for  ADS  implementation.  These  methodologies  are  being 
separately  developed  by  the  JADS  JTF  and  include; 

•  JADS  Special  Report  on  ADS  T&E  Methodologies  including; 

•  ADS  Test  Concept  Development  Methodology 

•  ADS  Test  Planning  and  Implementation  Methodology.  This  is  outlined  in  Appendix  C, 
ADS  Test  Implementation  Checklist,  and  illustrates  the  strong  system  engineering 
approach  needed  to  solve  problems. 

•  JADS  Special  Report  on  ADS  Cost  Guidance  including  overall  potential  benefits  and  cost 
savings  associated  with  the  use  of  ADS 

•  JADS  Special  Report  on  Verification,  Validation,  and  Accreditation  of  Distributed  Tests 

•  JADS  Special  Report  on  Networking  and  Engineering 

•  JADS  Special  Report  on  Distributed  Test  Control 

•  JADS  Special  Report  on  Programmatic  Challenges  to  ADS-Based  T&E 
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4.0  ADS-Supported  C4ISR  Testing  Applications 


This  section  considers  other  applications  of  ADS  to  C4ISR  testing  beyond  that  of  the  ETE  Test. 
C4ISR  systems  are  distributed  by  their  nature  and  obvious  candidates  for  distributed  testing. 
Since  all  C4ISR  systems  contain  the  same  basic  elements  (e.g.,  sensor(s),  sensor  platform(s), 
command  and  control  elements,  communication  lines,  and  computer  hardware  and  software),  the 
extension  of  the  ETE  Test  results  to  other  possible  C4ISR  applications  is  relatively 
straightforward. 

4.1  Common  Considerations  for  Element  Representation 

Human-in-the-Loop.  A  key  element  in  C4ISR  testing  is  the  evaluation  of  human-in-the-loop 
actions,  decision  processes,  and  timelines,  which  digital  simulations  do  not  model  well.  For  this 
reason,  ADS-supported  C4ISR  testing  should  include  live  participants  for  all  key  operators  and 
decision  makers.  If  available,  the  actual  equipment/tactical  workstations/command  posts  should 
be  employed  and  manned  by  trained  personnel.  If  actual  equipment  is  not  used,  the  human- 
machine  interface  can  be  represented  by  computer  displays  that  present  the  same  information  in 
the  same  format. 

Communications  Links.  When  the  actual  workstations  are  used  in  C4ISR  testing,  the  actual 
tactical  communication  links  and  message  protocols  may  be  used.  However,  geographical 
separation  between  C4ISR  system  elements  may  prevent  line-of-sight  communications.  In  this 
case,  land  lines  (i.e.,  T-1  links)  may  have  to  be  used,  but  the  messages  could  still  use  realistic 
tactical  protocols  and  communication  equipment.  For  example,  the  ETE  Test  used  land  lines  but 
employed  such  doctrinally  correct  communications  as  the  CGS-100  with  remote  workstations, 
SCDL  messages  between  the  ATWS  and  the  LGSM,  and  AFATDS-to-AFATDS  message  traffic. 

Sensor  and  Sensor  Platform.  There  are  different  considerations  depending  on  whether  the  sensor 
and  sensor  platform  representations  are  live  or  simulated. 

•  Live  Sensor/Platform.  Use  of  the  actual  sensor  and  sensor  platform  will  provide  real 
background,  clutter,  and  contrast  scenes  for  target  detection.  However,  the  full  battlefield 
array  of  potential  targets  generally  cannot  be  provided  from  live  entities.  Thus,  the  live 
targets  must  be  supplemented  with  virtual  entities  using  interface  hardware  and  software 
which  can  integrate  live  and  virtual  targets. 

•  The  preferred  location  of  the  interface  is  behind  the  sensor  and  before  any  sensor  signal 
processing  so  that  live  and  virtual  targets  are  detected  and  processed  in  the  same  manner. 
This  was  the  approach  used  in  the  ETE  Test;  virtual  targets  were  uplinked  to  the  aircraft 
and  injected  into  the  Joint  STARS  radar  scene  before  processing  and  display  to  the  E-8C 
ATWS  operators. 

•  In  some  applications,  it  may  be  impossible  or  unnecessary  to  locate  the  interface  on  the 
sensor  platform.  For  example,  with  an  unmanned  aerial  vehicle  (UAV)  sensor,  weight 
and  space  limitations  may  prevent  installation  of  the  interface  hardware  on  the  UAV. 


21 


However,  the  UAV  video  is  not  observed  on  the  platform  but  at  a  ground  station.  Thus, 
the  interface  could  be  located  at  the  UAV  ground  station. 

•  Simulated  Sensor/Platform.  If  the  sensor  and  sensor  platform  are  represented  by  simulations, 
then  the  scenario  will  also  be  provided  by  simulations,  and  the  interface  will  be  relatively 
straightforward.  The  actual  sensor  signal  processors  and  software  can  still  be  used  to  prepare 
the  sensor  output  for  the  actual  operator  displays,  thus  providing  realistic  testing  of  these 
subsystems. 

Battlefield  Scenario.  The  major  shortfall  in  current  C4ISR  developmental  and  operational  testing 
is  in  providing  realistic  battlefield  scenario  stimuli  to  the  sensor  subsystem.  By  using  accredited 
combat  simulations,  an  ADS-enhanced  synthetic  environment  can  allow  for  testing  that  would 
otherwise  be  impractical.  A  number  of  accredited  simulations  exist  for  representing  the 
battlefield  scenario,  and  the  choice  of  simulation  depends  on 

•  number  of  virtual  entities  needed, 

•  level  of  control  needed  over  the  simulation, 

•  need  to  interface  at  the  entity  or  the  aggregate  level, 

•  need  to  represent  a  battalion-,  brigade-  or  corps-level  scenario, 

•  size  of  the  terrain  box  required  for  the  C4ISR  test, 

•  extent  of  the  simulation’s  verification  and  validation  “pedigree,”  and 

•  compatibility  of  the  simulation  with  the  other  components  of  the  ADS  synthetic  environment. 

Weapon  System.  Although  not  all  C4ISR  tests  include  a  shooter,  mission-level  evaluations  of 
many  C4ISR  systems  require  evaluating  the  combat  effectiveness  impacts  of  the  system  in 
locating  and  identifying  enemy  targets  which  are  subsequently  destroyed.  Hence,  the  weapon 
system  must  also  be  represented  in  mission-level  evaluations.  Since  most  of  the  targets  in  the 
ADS-supported  test  are  simulated,  the  weapon  itself  will  normally  also  be  simulated,  although 
with  tailored  fidelity  according  to  test  requirements. 

4.2  Specific  Example  -  National  Missile  Defense 

As  an  example,  the  application  of  ADS  to  C4ISR  system  testing  involving  National  Missile 
Defense  (NMD)  scenarios  is  considered.  Because  of  the  large  number  of  variables,  safety 
concerns,  and  the  high  costs  associated  with  live  testing,  the  NMD  program  must  rely  heavily  on 
modeling  and  simulation  during  T&E.  Since  the  NMD  systems  will  be  operationally  distributed 
with  many  operators  and  decision  makers,  ADS-supported  testing  would  provide  much  of  the 
desired  operational  realism. 

As  summarized  in  Section  2.2,  distributed  testing  is  already  being  used  to  support  the  TMD 
program.  Under  the  sponsorship  of  the  Ballistic  Missile  Defense  Organization  (BMDO),  the 
TMDSE  at  the  JNTF  uses  DIS  PDUs  to  support  real-time,  operator-in-the-loop  testing  of  real, 
geographically  distributed  TMD  tactical  sensors  and  weapon  systems  that  are  linked  using  their 
respective  tactical  communication  data  nets  [Ref.  8].  Using  TMDSE  as  a  scenario  driver,  the 
feasibility  of  distributed  testing  of  TMD  family-of-systems  (FoS)  architectures  has  already  been 
demonstrated.  The  application  to  NMD  architectures  would  seem  to  be  straightforward. 
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The  NMD  program  has  used  the  Integrated  System  Test  Capability  (ISTC)  to  support  system- 
level  processor-in-the-loop  tests.  In  the  current  implementation,  individual  computer  stations  or 
nodes  within  the  Advanced  Research  Center  (ARC),  Huntsville,  Alabama,  will  represent  the 
various  elements  of  the  NMD  architecture  (e.g..  Battle  Management  Command,  Control,  and 
Communications  (BM/C3),  Upgraded  Early  Warning  Radar  (UEWR),  ground  based  radar 
(GBR),  Space  Based  Infrared  System  (SBIRS),  ground  based  interceptor  (GBI)).  These  nodes 
are  linked  together,  and  ISTC  supplies  them  with  the  test  scenario  (simulated  threats  including 
threat  signatures  and  kinematics  and  environments).  These  tests  use  the  actual  element  mission 
and  communications  processors  and  software,  if  available  (otherwise  the  element  is  represented 
by  a  digital  model),  but  there  are  no  human-in-the-loop  interactions  [Ref.  26]. 

There  is  a  proposal  to  use  ISTC  as  part  of  distributed  NMD  testing  by  incorporating  ISTC  into 
system  integrated  exercises  (SIEs)  [Ref.  26].  SIEs  are  system-level  distributed  tests,  and  ISTC 
would  be  used  as  the  test  scenario  driver.  The  NMD  element  representations  and  drivers  for  the 
SIEs  will  evolve  over  time  from  digital  models  to  actual  hardware  and  software  with  real 
operators.  In  particular,  the  SIEs  would  be  used  to  evaluate  the  NMD  BM/C3  capability  for  the 
designated  operational  commander  to  plan,  coordinate,  direct,  and  control  NMD  weapons  and 
sensors,  a  critical  issue  in  the  NMD  program. 

Distributed  tests  involving  the  use  of  a  scenario  driver,  such  as  ISTC,  along  with  the  actual  NMD 
hardware  and  software,  high-fidelity  hardware-in-the-loop,  tactical  operators-in-the-loop,  and 
tactical  communications  links  are  feasible  based  on  TMDSE  experience  and  would  provide 
realistic  testing  against  the  full  many-on-many  NMD  scenarios  that  cannot  be  used  in  live  tests. 

4.3  Specific  Example  -  Airborne  Laser 

The  Airborne  Laser  (ABL)  is  being  developed  as  an  antimissile  element  of  TMD  architectures. 
A  real-time,  operator-in-the-loop  ABL  simulation  has  been  developed  at  the  Theater  Air 
Command  and  Control  Simulation  Facility  (TACCSF),  Kirtland  Air  Force  Base,  New  Mexico 
[Ref.  27].  The  simulation  is  capable  of  interacting  in  distributed  exercises  which  provide  the 
ABL  System  Program  Office  with  a  unique  environment  for  testing  both  operational  and 
technical  concepts.  There  are  recommendations  for  using  linked  testing  to  form  a  robust 
capability  of  evaluate  overall  ABL  system  performance  in  a  joint  theater  environment  [Ref.  28]. 
Since  ADS  can  link  together  portions  of  the  ABL  system  and  TMD  architecture,  this  test 
capability  should  be  able  to  examine  all  functional  areas  at  the  engagement,  mission,  and  theater 
levels. 

Distributed  tests  involving  the  TACCSF  ABL  simulation  are  encouraged,  since  they  are  feasible 
and  would  provide  a  means  for  realistic  evaluation  of  the  ABL  at  the  mission  and  theater  levels. 
In  particular,  the  BM/C3  functional  area  can  be  addressed  at  the  theater  level  using  distributed 
testing. 
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5.0  Conclusion 


5.1  Summary 

The  findings  from  the  ETE  Test  support  the  conclusion  that  ADS  has  significant  utility  for 
C4ISR  DT&E  and  OT&E.  In  fact,  ADS  techniques  can  potentially  be  applied  to  all  C4ISR 
acquisition  phases.  ADS  is  an  enabling  technology  for  STEP  and  SBA,  and  ADS  is  useful  for 
supporting  the  iterative  spiral  development  process.  There  are  benefits  to  the  appropriate  use  of 
ADS  in  support  of  C4ISR  testing  including  its  use  as  a  viable  technique  for  performing  realistic 
mission-level  evaluations.  Constraints  and  concerns  for  implementing  ADS  are  more 
programmatic  and  cultural  than  technical. 

This  report  gives  general  guidelines  for  implementation  of  ADS-based  testing  of  C4ISR  systems. 
Detailed  requirements  for  linking  specific  C4ISR  systems  using  specific  architectures  were  not 
developed. 

The  ETE  Test  results  strongly  suggest  that  ADS  has  excellent  potential  for  improving  C4ISR 
testing  and  system  acquisition.  Thus,  test  planners  and  program  managers  should  consider  the 
technology  as  a  relevant  tool  for  their  program  unless  an  objective  assessment  suggests 
otherwise. 

5.2  Recommendations 

JADS  recommends  the  following  in  order  to  increase  the  utility  of  ADS  to  C4ISR  programs. 

•  Require  that  ADS  approaches  be  addressed  in  the  acquisition  strategy  and  the  test  and 
evaluation  master  plan  (TEMP)  for  new  C4ISR  acquisition  programs. 

•  Direct  all  programs  to  focus  on  the  ability  of  ADS  to  overcome  any  identified  test  limitations. 

•  When  making  an  ADS  go/no  go  decision,  compare  ADS  costs  and  benefits  to  those  of  the 
alternative  method(s). 

•  Develop  infrastructure  to  reduce  the  costs  of  linking. 

•  Use  an  ADS  environment  over  the  life  of  a  program. 

•  Use  JADS -developed  ADS  methodologies,  e.g.,  test  planning,  verification  and  validation, 
ADS  implementation. 

•  Institute  ADS  training  at  formal  T&E  and  acquisition  schools. 

•  Nurture  ground-breaking  programs,  e.g.,  FI  2010,  Joint  Strike  Fighter. 

•  Subsidize  service  ADS  pilot  projects  on  C4ISR  acquisition  programs. 

•  Use  ADS  to  enhance  real-time  data  analysis  and  test  efficiency. 

•  Develop  inherent  linking  capability  at  facilities  and  ranges. 

•  Provide  on-call  WAN  access. 

•  Develop  robust  standards  for  interoperability  of  both  DSMs  and  test  ranges  and  facilities. 
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Appendix  A  -  List  of  Acronyms 


ABL 

ADS 

AFATDS 

AFOTEC 

AO 

ARC 

ASAS 

ATACMS 

ATWS 

BM/C3 

BMDO 

C2 

C3 

C3I 

C4I 

C4ISR 

CAMPS 

CGS 

ConOps 

CSU 

CTEIP 

DD,  DT&E/S&TS 

DIS 

DMAP 

DMSO 

DoD 

DoDD 

DREN 

DSI 

DSM 

DSU 

DT&E 

ETE 

EW 

FBCB2 

FDC 

FED 

FEDEP 

n2010 


Airborne  Laser 

advanced  distributed  simulation 
Advanced  Field  Artillery  Tactical  Data  System 
Air  Force  Operational  Test  and  Evaluation  Center 
attack  operations 

Advanced  Research  Center,  Huntsville,  Alabama 
All  Source  Analysis  System 
Army  Tactical  Missile  System 
Advanced  Technology  Work  Station 

Battle  Management  Command,  Control,  and  Communications 

Ballistic  Missile  Defense  Organization 

command  and  control 

command,  control,  and  communications 

command,  control,  communications,  and  intelligence 

command,  control,  communications,  computers  and  intelligence 

command,  control,  communications,  computers,  intelligence,  surveillance 

and  reconnaissance 

Compartmented  All  Source  Analysis  System  (ASAS)  Message  Processing 
System 

common  ground  station 
concept  of  operation 
channel  service  unit 

Central  Test  and  Evaluation  Investment  Program 

Deputy  Director,  Developmental  Test  and  Evaluation,  Strategic  and 

Tactical  Systems 

distributed  interactive  simulation 

data  management  and  analysis  plan 

Defense  Modeling  and  Simulation  Organization,  Alexandria,  Virginia 

Department  of  Defense 

Department  of  Defense  directive 

Defense  Research  and  Engineering  Network 

Defense  Simulation  Internet 

digital  system  model 

data  service  unit 

developmental  test  and  evaluation 

end-to-end 

electronic  warfare 

Force  XXI  Battle  Command  Brigade  and  Below 

fire  direction  center 

federation 

federation  development  and  execution  process 
Foundation  Initiative  2010 
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FOM 

FoS 

GBI 

GBR 

HLA 

ICD 

IEEE 

ISG 

ISTC 

ITEA 

JADS 

Janus 

JNTF 

Joint  STARS 
JSIMS 
JT&E 
JTF 

JTMD  AO 

LGSM 

LOS 

M&S 

NMD 

OPTEMPO 

OSD 

OT&E 

PDU 

RTI 

SAIC 

SATCOM 

SBA 

SBIRS 

SCDL 

SIE 

SIPRNET 

SISO 

SIT 

SIW 

STEP 

STORM 

STOW 

T«feE 

T-1 

TAG 

TACCSF 


federation  object  model 

family  of  systems 

ground  based  interceptor 

ground  based  radar 

high  level  architecture 

interface  control  document 

Institute  of  Electrical  and  Electronics  Engineers 

Implementation  Study  Group 

integrated  system  test  capability 

International  Test  and  Evaluation  Association 

Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 

interactive,  computer-based  simulation  of  combat  operations 

Joint  National  Test  Facility,  Falcon  Air  Force  Base,  Colorado 

Joint  Surveillance  Target  Attack  Radar  System 

Joint  Simulation  System 

joint  test  and  evaluation 

joint  test  force 

Joint  Theater  Missile  Defense  Attack  Operations 

light  ground  station  module 

line-of-sight 

modeling  and  simulation 

National  Missile  Defense 

operations  tempo 

Office  of  the  Secretary  of  Defense 

operational  test  and  evaluation 

protocol  data  unit 

runtime  infrastmcture 

Science  Applications  International  Corporation 

satellite  communications 

Simulation  Based  Acquisition 

Space  Based  Infrared  System 

surveillance  and  control  data  link 

system  integrated  exercise 

Secure  Internet  Protocol  Router  Network 

Simulation  Interoperability  Standards  Organization 

system  integration  test 

simulation  interoperability  workshop 

Simulation,  Test,  and  Evaluation  Process 

Simulation,  Testing,  Operations,  Rehearsal  Model 

Synthetic  Theater  of  War 

test  and  evaluation 

digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1.544  megabits 
per  second 
target  analysis  cell 

Theater  Air  Command  and  Control  Simulation  Facility 
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TAFSM 

Tactical  Army  Fire  Support  Model 

TCAC 

Test  Control  and  Analysis  Center,  Albuquerque,  New  Mexico 

TECOM 

Test  and  Evaluation  Command 

TEMP 

test  and  evaluation  master  plan 

TENA 

Test  and  Training  Enabling  Network  Architecture 

TMD 

Theater  Missile  Defense 

TMDSE 

Theater  Missile  Defense  System  Exerciser 

TRAC 

U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Analysis  Center 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

TSPI 

time-space-position  information 

UAV 

unmanned  aerial  vehicle 

UEWR 

Upgraded  Early  Warning  Radar 

VPG 

virtual  proving  ground 

VSTARS 

Virtual  Surveillance  Target  Attack  Radar  System 

W&A 

verification,  validation  and  accreditation 

WAN 

wide  area  network 

WSMR 

White  Sands  Missile  Range 
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Appendix  B  -  Glossary 


A 

Accreditation.  See:  distributed  simulation  accreditation,  model/simulation  accreditation. 

Accuracy.  The  degree  of  exactness  of  a  model  or  simulation  relative  to  an  established  standard 
with  high  accuracy  implying  low  error.  [DIS] 

Activity.  An  event  that  consumes  time  and  resources  and  whose  performance  is  necessary  for  a 
system  to  move  from  one  event  to  the  next.  [DIS] 

Advanced  Distributed  Simulation  (ADS).  A  set  of  disparate  models  or  simulations  operating 
in  a  common  synthetic  environment.  The  ADS  may  be  composed  of  three  modes  of 
simulation:  live,  virtual  and  constructive,  where  the  latter  can  be  seamlessly  integrated  within 
a  single  exercise.  See  also:  live  simulation;  virtual  simulation;  constmctive  simulation. 

[DIS] 

Aggregate.  An  activity  that  combines  individual  entities  into  a  singular  entity.  Contrast  with: 
disaggregate.  [DIS] 

B 

Battlespace.  The  three-dimensional  battlefield.  [DIS] 

Benchmark,  (v)  The  activity  of  comparing  the  results  of  a  model  or  simulation  with  an  accepted 
representation  of  the  process  being  modeled,  (n)  The  accepted  representation  of  the  modeled 
process.  [DIS] 

Bit.  The  smallest  unit  of  information  in  the  binary  system  of  notation.  [IEEE  1278.1] 

Broadcast.  A  transmission  mode  in  which  a  single  message  is  sent  to  all  network  destinations, 
i.e.,  one-to-all.  Broadcast  is  a  special  case  of  multicast.  Contrast  with:  multicast;  unicast. 
[IEEE  1278.2] 

c 

Compatible.  Two  or  more  simulations  are  distributed  interactive  simulation  (DIS)  compatible  if 
(1)  they  are  DIS  compliant,  and  (2)  their  models  and  data  that  send  and  interpret  protocol  data 
units  (PDUs)  support  the  realization  of  a  common  operational  environment  among  the 
systems  (coherent  in  time  and  space).  Contrast  with:  compliant,  interoperable.  [DIS] 

Compliant.  A  simulation  is  distributed  interactive  simulation  (DIS)  compliant  if  it  can  send  or 
receive  protocol  data  units  (PDUs)  in  accordance  with  Institute  of  Electrical  and  Electronics 
Engineers  (IEEE)  Standard  1278  and  1278  (working  drafts).  A  specific  statement  must  be 
made  regarding  the  qualifications  of  each  PDU.  Contrast  with:  compatible,  interoperable. 
[DIS] 

Conceptual  Model.  A  description  of  the  content  and  internal  representations  which  are  the 
user's  and  developer's  combined  concepts  of  the  exercise.  It  includes  logic  and  algorithms 
and  explicitly  recognizes  assumptions  and  limitations.  [DIS] 
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Constructive  Simulation.  Models  and  simulations  that  involve  simulated  people  operating 
simulated  systems.  See  Also:  war  games;  higher  order  model  (HOM).  [DIS] 

Continuous  Model.  (1)  A  mathematical  or  computational  model  whose  output  variables  change 
in  a  continuous  manner;  that  is,  in  changing  from  one  value  to  another,  a  variable  can  take  on 
all  intermediate  values.  For  example,  a  model  depicting  the  rate  of  air  flow  over  an  airplane 
wing.  Syn:  continuous-variable  model.  (2)  A  model  of  a  system  that  behaves  in  a  continuous 
manner.  Contrast  with:  discrete  model.  [DIS] 

Continuous  Simulation.  A  simulation  that  uses  a  continuous  model.  [DIS] 
Continuous-Variable  Model.  See:  continuous  model.  [DIS] 

Control  Station.  (1)  A  facility  which  provides  the  individual  responsible  for  controlling  the 
simulation  and  the  capability  to  implement  simulation  control  as  protocol  data  units  (PDUs) 
on  the  distributed  interactive  simulation  (DIS)  network. 

Syn:  simulation  -  management  station.  [DIS] 

D 

Data.  Representation  of  facts,  concepts,  or  instructions  in  a  formalized  manner  suitable  for 
communication,  interpretation  or  processing  by  humans  or  automatic  means.  [DIS] 

Database.  A  collection  of  data  organized  according  to  a  schema  to  serve  one  or  more 
applications.  [DIS] 

Data  Certification.  The  determination  that  data  have  been  verified  and  validated.  (1)  Data 
producer  certification  is  the  determination  by  the  data  producer  that  data  have  been  verified 
and  validated  against  documented  standards  of  criteria.  (2)  Data  user  certification  is  the 
determination  by  the  application  sponsor  or  designated  agent  that  data  have  been  verified  and 
validated  as  appropriate  for  the  specific  modeling  and  simulation  (M&S)  usage.  [DIS] 

Data  Logger.  A  device  that  accepts  protocol  data  units  (PDUs)  from  the  network  and  stores 
them  for  later  replay  in  the  same  time  sequence  as  the  PDUs  were  originally  received.  See 
also:  protocol  data  unit  (PDU).  [IEEE  1278.3] 

Data  Validation.  The  documented  assessment  of  data  by  subject  area  experts  and  its 
comparison  to  known  or  best-estimate  values.  (1)  Data  producer  validation  is  that 
documented  assessment  within  stated  criteria  and  assumptions.  (2)  Data  user  validation  is 
that  documented  assessment  of  data  as  appropriate  for  use  in  an  intended  modeling  and 
simulation  (M&S).  [DIS] 

Data  Verification.  The  use  of  techniques  and  procedures  to  ensure  that  data  meet  specified 
constraints  defined  by  data  standards  and  business  mles.  (1)  Data  producer  verification  is  the 
use  of  techniques  and  procedures  to  ensure  that  data  meet  constraints  defined  by  data 
standards  and  business  rules  derived  from  process  and  data  modeling.  (2)  Data  user 
verification  is  the  use  of  techniques  and  procedures  to  ensure  that  data  meet  user  specified 
constraints  defined  by  data  standards  and  business  rules  derived  from  process  and  data 
modeling  and  that  data  are  transformed  and  formatted  properly.  [DIS] 
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Data  Verification,  Validation,  and  Certification.  The  process  of  verifying  the  internal 
consistency  and  correctness  of  data,  validating  that  they  represent  real-world  entities 
appropriate  for  their  intended  purpose  or  an  expected  range  of  purposes,  and  certifying  them 
as  having  a  specified  level  of  quality  or  as  being  appropriate  for  a  specified  use,  type  of  use, 
or  range  of  uses.  The  process  has  two  perspectives:  producer  and  user  process.  See:  data 
validation,  data  verification,  and  data  certification.  [DIS] 

Dead  Reckoning.  See:  remote  entity  approximation. 

Deaggregate.  See:  disaggregate.  [DIS] 

Distributed  Interactive  Simulation  (DIS).  A  synthetic  environment  within  which  humans  may 
interact  through  simulation(s)  at  multiple  sites  networked  using  compliant  architecture, 
protocols,  standards,  and  databases  (DoDD  5000.59P) 

E 

Electronic  Battlefield.  See:  synthetic  environment.  [DIS] 

Entity.  Any  component  in  a  system  that  requires  explicit  representation  in  a  model.  Entities 
possess  attributes  denoting  specific  properties.  See:  simulation  entity.  [DIS] 

Environment.  (1)  The  texture  or  detail  of  the  domain,  such  as  cities,  farmland,  sea  states,  etc. 

(2)  The  external  objects,  conditions,  and  processes  that  influence  the  behavior  of  a  system 
(such  as  terrain  relief,  weather,  day,  night,  terrain  cultural  features,  etc.)  [DIS] 

Event.  (1)  An  occurrence  that  causes  a  change  of  state  in  a  simulation.  See  also:  conditional 
event;  time-dependent  event.  (2)  The  instant  in  time  at  which  a  change  in  some  variable 
occurs.  [DIS] 

Event-Driven  Simulation.  See:  event-oriented  simulation.  [DIS] 

Event-Oriented  Simulation.  A  simulation  in  which  attention  is  focused  on  the  occurrence  of 
events  and  the  times  at  which  those  events  occur;  for  example,  a  simulation  of  a  digital 
circuit  that  focuses  on  the  time  of  state  transition.  Syn:  event-driven  simulation;  event- 
sequenced  simulation.  [DIS] 

Event-Sequenced  Simulation.  See:  event-oriented  simulation.  [DIS] 

Exercise.  (1)  One  or  more  sessions  with  a  common  objective  and  accreditation.  (2)  The  total 
process  of  designing,  assembling,  testing,  conducting,  evaluating,  and  reporting  on  an 
activity.  See:  simulation  exercise.  Syn:  experiment,  demonstration.  [DIS,  IEEE  1278.3] 

F 

Fidelity.  (1)  The  similarity,  both  physical  and  functional,  between  the  simulation  and  that  which 
it  simulates.  (2)  A  measure  of  the  realism  of  a  simulation.  (3)  The  degree  to  which  the 
representation  within  a  simulation  is  similar  to  a  real-world  object,  feature,  or  condition  in  a 
measurable  or  perceivable  manner.  See  also:  model/simulation  validation.  [DIS,  IEEE 
1278.1] 

Field.  (1)  A  series  of  contiguous  bits,  treated  as  an  instance  of  a  particular  data  type,  that  may  be 
part  of  a  higher  level  data  structure.  (2)  An  external  operating  area  for  actual  vehicles  or  live 
entities.  See:  field  instrumentation.  [DIS,  IEEE  1278.1] 
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G 

Graphical  Model.  A  symbolic  model  whose  properties  are  expressed  in  diagrams.  For 

example,  a  decision  tree  used  to  express  a  complex  procedure.  Contrast  with:  mathematical 
model;  narrative  model;  software  model;  tabular  model.  [DIS] 

Ground  Truth.  The  actual  facts  of  a  situation  without  errors  introduced  by  sensors  or  human 
perception  and  judgment.  [DIS] 

H 

Human-in-the-Loop  Model.  See:  interactive  model. 

Human-Machine  Simulation.  A  simulation  carried  out  by  both  human  participants  and 
computers,  typically  with  the  human  participants  asked  to  make  decisions  and  a  computer 
performing  processing  based  on  those  decisions.  [DIS] 

I 

Interactive  Model.  A  model  that  requires  human  participation.  Syn:  human-in-the-loop  model. 
[DIS] 

Interoperable.  Two  or  more  simulations  are  distributed  interactive  simulation  (DIS) 
interoperable  for  a  given  exercise  if  they  are  DIS  compliant,  DIS  compatible,  and  their 
performance  characteristics  support  a  fair  fight  to  the  fidelity  required  for  the  exercise. 
Contrast  with:  compatible,  compliant.  [DIS] 

Interoperability.  (1)  The  ability  of  a  set  of  simulation  entities  to  interact  with  an  acceptable 
degree  of  fidelity.  The  acceptability  of  a  model  is  determined  by  the  user  for  the  specific 
purpose  of  the  exercise,  test,  or  analysis.  (2)  The  ability  of  a  set  of  distributed  interactive 
simulation  applications  to  interact  through  the  exchange  of  protocol  data  units.  [DIS] 

L 

Live  Entity.  A  perceptible  object  that  can  appear  in  the  virtual  battlespace  but  is  unaware  and 
nonresponsive  (either  by  intent,  lack  of  capability  or  circumstance)  to  the  actions  of  virtual 
entities.  See  also:  field  instmmentation.  Contrast  with:  live  instrumented  entity.  [DIS] 

Live  Instrumented  Entity.  A  physical  entity  that  is  in  the  real  world  and  can  be  represented  in 
the  distributed  interactive  simulation  (DIS)  virtual  battlespace  which  can  be  manned  or 
unmanned.  The  live  instrumented  entity  has  internal  and/or  external  field  instrumentation 
(FI)  devices/systems  to  record  and  relay  the  entity’s  surroundings,  behavior,  and/or  reaction 
to  events.  If  the  FI  provides  a  two-way  link,  the  events  that  affect  the  live  instrumented  entity 
can  be  occurring  in  the  virtual  battlespace  as  well  as  the  real  world.  See  also:  field 
instrumentation,  live  entity.  [DIS] 

Local  Area  Network  (LAN).  A  class  of  data  network  which  provides  high  data  rate 
interconnection  between  network  nodes  in  close  physical  proximity.  [IEEE  1278.3] 
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M 

Measure  of  Performance  (MOP).  Measure  of  how  the  system/individual  performs  its  functions 
in  a  given  environment  (e.g.,  number  of  targets  detected,  reaction  time,  number  of  targets 
nominated,  susceptibility  of  deception,  task  completion  time).  It  is  closely  related  to  inherent 
parameters  (physical  and  structural)  but  measures  attributes  of  system  behavior.  See  also; 
measures  of  effectiveness  (MOE).  [lEE  1278.3] 

Model.  (1)  An  approximation,  representation,  or  idealization  of  selected  aspects  of  the  structure, 
behavior,  operation,  or  other  characteristics  of  a  real-world  process,  concept,  or  system.  Note: 
Models  may  have  other  models  as  components.  (2)  To  serve  as  a  model  as  in  (1).  (3)  To 
develop  or  use  a  model  as  in  (1).  (4)  A  mathematical  or  otherwise  logical  representation  of  a 
system  or  a  system’s  behavior  over  time.  [DIS] 

Model/Simulation  Accreditation.  The  official  certification  that  a  model  or  simulation  is 
acceptable  for  use  for  a  specific  purpose.  See  also:  distributed  simulation  accreditation. 
Contrast  with:  model/simulation  validation,  model/simulation  verification.  [DoDD  5000.59] 

Model/Simulation  Validation.  The  process  of  determining  the  degree  to  which  a  model  is  an 
accurate  representation  of  the  real  world  from  the  perspective  of  the  intended  use(s)  of  the 
model.  See  also:  distributed  simulation  validation,  fidelity.  Contrast  with:  model 
simulation  accreditation,  model  simulation  verification.  [DoDD  5000.59] 

Model/Simulation  Verification.  The  process  of  determining  that  a  model  implementation 
accurately  represents  the  developer's  conceptual  description  and  specifications.  See  also: 
distributed  simulation  verification.  Contrast  with:  model  simulation  accreditation,  model 
simulation  validation.  [DoDD  5000.59] 

N 

Network  Filter.  A  system  to  selectively  accept  or  reject  data  received  from  the  network.  [DIS] 

Network  Node.  A  specific  network  address.  See:  node.  Contrast  with:  processing  node.  [DIS] 

Node.  A  general  term  denoting  either  a  switching  element  in  a  network  or  a  host  computer 
attached  to  a  network.  See:  processing  node;  network  node.  [IEEE  1278.1,  IEEE  1278.2] 

o 

Operational  Environment.  A  composite  of  the  conditions,  circumstances,  and  influences 
which  affect  the  employment  of  military  (or  other)  forces  and  the  decisions  of  the  unit 
commander  or  person  in  charge.  [DIS] 

P 

Platform.  A  generic  term  used  to  describe  a  level  of  representation  equating  to  vehicles,  aircraft, 
missiles,  ships,  fixed  sites,  etc.,  in  the  hierarchy  of  representation  possibilities.  Other 
representation  levels  include  units  (made  up  of  platforms)  and  components  or  modules 
(which  make  up  platforms.)  [DIS] 
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Protocol  Data  Unit  (PDU).  A  distributed  interactive  simulation  (DIS)  data  message  that  is 
passed  on  a  network  between  simulation  applications  according  to  a  defined  protocol.  [IEEE 
1278.1] 

R 

Real  Time.  In  modeling  and  simulation,  simulated  time  advances  at  the  same  rate  as  actual  time; 
for  example,  running  the  simulation  for  one  second  results  in  the  model  advancing  time  by 
one  second.  Contrast  with:  fast  time,  slow  time.  [DIS] 

Resolution.  (1)  The  degree  to  which  near  equal  results  values  can  be  discriminated.  (2)  The 
measure  of  the  ability  to  delineate  picture  detail.  [DIS] 

s 

Scenario.  (1)  Description  of  an  exercise  (initial  conditions).  It  is  part  of  the  session  database 
which  configures  the  units  and  platforms  and  places  them  in  specific  locations  with  specific 
missions.  (2)  An  initial  set  of  conditions  and  time  line  of  significant  events  imposed  on 
trainees  or  systems  to  achieve  exercise  objectives.  See:  field  exercise.  [DIS,  IEEE  1278.3] 

SIMNET  (Simulator  Networking).  The  prototype  distributed  simulation  upon  which 
distributed  interactive  simulation  (DIS)  was  based.  [DIS] 

Simulate.  To  represent  a  system  by  a  model  that  behaves  or  operates  like  the  system.  See  also: 

•  emulate.  [DIS] 

Simulated  Time.  Time  as  represented  within  a  simulation.  Syn:  virtual  time.  See  also:  fast 
time;  real  time;  slow  time.  [DIS] 

Simulation.  (1)  A  model  that  behaves  or  operates  like  a  given  system  when  provided  a  set  of 
controlled  inputs.  Syn:  simulation  model.  See  also:  emulation.  (2)  The  process  of 
developing  or  using  a  model  as  in  (1).  (3)  An  implementation  of  a  special  kind  of  model  that 
represents  at  least  some  key  internal  elements  of  a  system  and  describes  how  those  elements 
interact  over  time.  [DIS] 

Simulation  Environment.  (1)  Consists  of  the  natural  physical  environment  surrounding  the 
simulation  entities  including  land,  oceans,  atmosphere,  near-space,  and  cultural  information. 
(2)  All  the  conditions,  circumstances,  and  influences  surrounding  and  affecting  simulation 
entities  including  those  stated  in  (1).  [DIS] 

’  Simulation  Exercise.  An  exercise  that  consists  of  one  or  more  interacting  simulation 

applications.  Simulations  participating  in  the  same  simulation  exercise  share  a  common 
identifying  number  called  the  exercise  identifier.  These  simulations  also  utilize  correlated 
representations  of  the  synthetic  environment  in  which  they  operate.  See:  live  simulation. 
[IEEE  1278.1,  IEEE  1278.2] 

Simulation  Fidelity.  Refers  to  the  degree  of  similarity  between  the  simulated  situation  and  the 
operational  situation.  [IEEE  1278.3] 

Simulation  Time.  (1)  A  simulation’s  internal  representation  of  time.  Simulation  time  may 
accumulate  faster,  slower,  or  at  the  same  pace  as  real  time.  (2)  The  reference  time  (e.g., 
universal  coordinated  time)  within  a  simulation  exercise.  This  time  is  established  ahead  of 
time  by  the  simulation  management  function  and  is  common  to  all  participants  in  a  particular 
exercise.  [DIS,  IEEE  1278.1] 
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Simulator.  (1)  A  device,  computer  program,  or  system  that  performs  simulation.  (2)  For 

training,  a  device  which  duplicates  the  essential  features  of  a  task  situation  and  provides  for 
direct  practice.  (3)  For  distributed  interactive  simulation  (DIS),  a  physical  model  or 
simulation  of  a  weapons  system,  set  of  weapon  systems,  or  piece  of  equipment  which 
represents  some  major  aspects  of  the  equipment’s  operation.  [DIS] 

Site.  (1)  An  actual  physical  location  at  a  specific  geographic  area,  e.g.,  the  Fort  Knox  Close 
Combat  Test  Bed  (CCTB).  (2)  A  node  on  the  network  used  for  distributed  simulation  such  as 
the  Defense  Simulation  Internet  (DSI)  long  haul  network.  (3)  A  level  of  configuration 
authority  within  a  distributed  interactive  simulation  (DIS)  exercise.  [DIS] 

V 

Validation.  See:  data  validation,  distributed  simulation  validation,  face  validation, 
model/simulation  validation.  [DIS] 

Verification.  See:  data  verification,  distributed  simulation  verification,  model/simulation 
verification 

Verification  and  Validation  (V&V)  Proponent.  The  agency  responsible  for  ensuring  V&V  is 
performed  on  a  specific  model  or  simulation.  [DIS] 

Vignette.  A  self-contained  portion  of  a  scenario.  [DIS] 

Virtual  Battlespace.  The  illusion  resulting  from  simulating  the  actual  battlespace.  [DIS] 

w 

War  Game.  A  simulation  game  in  which  participants  seek  to  achieve  a  specified  military 

objective  given  pre-established  resources  and  constraints;  for  example,  a  simulation  in  which 
participants  make  battlefield  decisions  and  a  computer  determines  the  results  of  those 
decisions.  See  also:  management  game.  Syn:  constructive  simulation;  higher  order  model 
(HOM).  [DIS] 

Wide  Area  Network  (WAN).  A  communications  network  of  devices  which  are  separated  by 
substantial  geographical  distance.  Syn:  long  haul  network.  [IEEE  1278.3] 
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Appendix  C  -  ADS  Test  Implementation  Checklist 


This  appendix  outlines  useful  procedures  in  implementing  ADS-based  testing  based  on  steps 
given  in  the  HLA  Federation  Development  and  Execution  Process  (FEDEP)  Model^.  The  more 
detailed  version  of  this  checklist  is  available  from  the  JADS  JTF  as  the  “Special  Report  on 
Advanced  Distributed  Simulation-Inclusive  Test  and  Evaluation  Methodologies.”^ 

STEP  1:  Define  Distributed  Test  Objectives 

Activity  1.1:  Identify  Needs 

-  Activity  Purpose:  develop  clear  understanding  of  the  problem  to  be  addressed  by  the 
distributed  test. 

-  Activity  Inputs:  program  objectives,  test  requirements,  and  information  on  resources 
available  to  support  a  distributed  test. 

-  Activity  Output:  needs  statement. 

Activity  1.2:  Develop  Objectives 

-  Activity  Purpose:  refine  the  needs  statement  into  a  more  detailed  set  of  specific 
objectives  for  the  distributed  test. 

-  Activity  Inputs:  needs  statement  from  previous  activity. 

-  Activity  Outputs:  statement  of  the  test  objectives  and  initial  planning  documents. 

STEP  2:  Develop  Conceptual  Model 

Activity  2.1:  Develop  Scenario 

-  Activity  Purpose:  develop  a  functional  specification  of  the  test  scenario. 

-  Activity  Inputs:  operational  context  constraints  specified  in  the  test  objectives  statement. 

-  Activity  Output:  test  scenario  description. 

Activity  2.2:  Perform  Conceptual  Analysis 

-  Activity  Purpose:  produce  a  conceptual  model  of  the  ADS  environment. 

-  Activity  Inputs:  test  scenario  description  from  previous  activity,  test  objectives 
statement,  and  any  doctrine  and  tactics  appropriate  for  the  scenario. 

-  Activity  Output:  conceptual  model,  which  is  a  description  of  the  players,  their  actions, 
and  any  interactions  among  players  that  need  to  be  included  in  the  distributed  test  in  order 
to  achieve  all  test  objectives. 

Activity  2.3:  Develop  Distributed  Test  Requirements 

-  Activity  Purpose:  define  top-level  test  requirements. 

-  Activity  Input:  conceptual  model  from  previous  activity. 

-  Activity  Output:  (1)  requirements  that  are  based  on  the  distributed  test  objectives,  are 
directly  testable,  and  provide  the  implementation  level  guidance  needed  to  design  and 


*  “High  Level  Architecture  Federation  Development  and  Execution  Process  (FEDEP)  Model,  Version  1.3,”  9 
December  1998  available  from  the  Defense  Modeling  and  Simulation  Organization  (DMSO)  HLA  Web  site  located 
at  http;//hla.dmso.mil/. 

9 

To  be  available  at  http://www.jads.abqxom  on  or  about  30  September  1999.  (After  1  March  2000  refer  requests  to 
HQ  AFOTEC/HO,  8500  Gibson  Blvd.  SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558  or  the  SAIC 
Technical  Library,  2001  North  Beauregard  St.  Suite  800,  Alexandria,  Virginia  22311.) 
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develop  the  distributed  test,  and  (2)  requirements-based  criteria  for  evaluating  test  results. 
Major  top-level  requirements  that  should  be  addressed  include  the  following: 

~  Fidelity  requirements  for  all  players  represented  in  the  test  scenarios. 

—  Interaction  requirements  specifying  information  types  that  must  be  exchanged  among 
players  to  permit  interactions. 

—  Latency  requirements  for  the  maximum  acceptable  latency  for  each  pair  of  interacting 
players. 

—  Data  reliability  requirements  specifying  the  maximum  acceptable  level  of  ADS- 
induced  errors,  such  as  dropout  rate  and  out-of-order  messages. 

~  Data  analysis  requirements  including  a  data  management  and  analysis  plan  (DMAP) 
detailing  the  analysis  approach  for  each  test  objective. 

STEP  3:  Design  and  Develop  Distributed  Test 

Activity  3. 1:  Design  ADS  Architecture 

-  Activity  Purpose:  establish  the  player  representations  in  the  distributed  test  and  develop  a 
coordinated  plan  to  guide  its  development,  test,  and  execution. 

-  Activity  Inputs:  test  requirements,  test  scenario,  conceptual  model,  and  the  initial 
planning  documents. 

-  Activity  Outputs:  identification  of  specific  player  representations  selected  and  a  detailed 
test  plan.  The  steps  for  this  activity  are  as  follows. 

—  Select  specific  player  representations. 

-  Develop  additional  requirements  needed  for  architecture  design  including: 

—  Data  requirements  including  data  rates,  time-space-position  information  (TSPI) 
accuracy  and  smoothness,  data  time-stamp  accuracy,  and  classification  of  the  data 
and  any  security  handling  procedures. 

—  Data  synchronization  requirements. 

—  Real-time  data  processing  requirements. 

—  Terrain  database  requirements. 

—  Data  collection/instrumentation  requirements. 

—  Post-test  data  management  requirements. 

—  Interface  requirements  to  include  simulation  interfaces,  special-purpose  interfaces, 
and  interfaces  to  the  runtime  infrastmcture  (RTI)  for  HLA  implementation. 

—  Test  control  and  monitoring  requirements. 

—  Determine  if  modifications  to  the  selected  player  representations  are  needed,  such  as: 
—  Simulation  modifications  needed  to  utilize  external  inputs. 

—  Simulation  modifications  needed  to  generate  the  required  outputs. 

—  Data  processing  modifications  needed  to  meet  TSPI  accuracy,  smoothness,  and 
latency  requirements. 

—  Facility  modifications  needed  for  a  replay  capability  that  can  be  used  during 
integration  testing. 

-  Determine  network  design  requirements  including  the  following: 

—  Data  protocols  to  be  used. 

—  Decide  if  HLA  will  be  implemented. 

—  Determine  if  data  from  each  node  will  be  broadcast,  multicast,  or  unicast 
(transmitted  point-to-point). 
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—  Determine  if  data  are  to  be  transmitted  using  best  effort  or  reliable  procedures. 

—  Determine  network  security  approach  to  be  implemented. 

—  Determine  the  wide  area  network  (WAN)  bandwidth  requirement. 

—  Prepare  a  detailed  plan  including  a  detailed  DMAP,  verification,  validation  and 
accreditation  (W&A)  plan,  and  an  interface  control  document  (ICD). 

Activity  3.2:  Develop  ADS  Architecture 

Activity  Purpose:  develop  the  federation  object  model  (FOM)  if  HLA  is  to  be 
implemented,  modify  the  simulations/range  facilities,  if  necessary,  and  prepare  the 
distributed  architecture  for  integration  and  test. 

-  Activity  Inputs:  detailed  test  plan  and  specific  player  representations  identified  in  the 
previous  activity. 

-  Activity  Outputs:  FOM  and  federation  execution  data  (FED)  file  if  HLA  is  to  be 
implemented  and  the  modified  player  representations.  The  ADS  architecture  is 
developed  during  this  activity  using  the  following  steps: 

--  Conduct  surveys  of  each  site  to  be  linked  by  the  network. 

—  Select  and  procure  the  WAN  to  be  used  to  link  the  nodes/facilities. 

—  Select  and  procure  the  network  hardware,  including  routers,  channel  service  units 
(CSUs)/data  service  units  (DSUs),  muliplexers,  encryptors,  etc. 

~  Select  and  procure  test  control  hardware  and  software. 

—  Select  and  procure  and/or  develop  network  analysis/monitoring  tools. 

~  Develop  detailed  procedures  for  secure/encrypted  operations  and  obtain  approval  for 
their  use  from  the  designated  approval  authority. 

STEP  4:  Integrate  and  Test  Architecture 

Activity  4.1:  Plan  Execution 

-  Activity  Purpose:  define  and  develop  the  full  set  of  information  required  to  support  the 
distributed  test  execution. 

-  Activity  Inputs:  FOM  and  FED  file  if  HLA  is  to  be  implemented  and  the  detailed  test 
plan. 

-  Activity  Outputs:  refined  and  detailed  integration  test  plan,  VV&A  plan,  and  test 
procedures.  The  integration  test  plan  should  address  the  following: 

—  Procedures  for  verifying  any  simulation/range  facility  modification  in  a  systematic 
stand-alone  fashion. 

-  Procedures  for  initially  testing  each  WAN  link  separately. 

—  Procedures  for  testing  each  simulation-to-simulation  connection  with  all  network 
nodes  connected. 

—  Procedures  for  testing  the  voice  communications  with  all  equipment  and  personnel  as 
in  the  actual  test. 

Activity  4.2:  Integrate  and  Test  ADS  Architecture 

-  Activity  Purpose:  bring  all  the  distributed  test  participants  into  a  unifying  operating 
environment  and  verify  that  they  can  all  interoperate  to  the  degree  required  to  achieve 
the  test  objectives. 

-  Activity  Inputs:  detailed  test  plan,  VV&A  plan,  and  test  procedures. 
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-  Activity  Outputs:  refined  test  procedures,  VV&A  results,  and  an  ADS  architecture 
which  has  been  thoroughly  tested  and  is  ready  for  test  execution.  Key  steps  during 
this  activity  include  the  following: 

-  Implement  the  simulation/range  facility  modifications  needed  for  linking,  as 
determined  during  Activity  3.1. 

—  Build/procure  the  necessary  interfaces  for  linking  in  accordance  with  the 
requirements  developed  during  Activity  3.1. 

—  Install  the  network  hardware  and  software. 

—  Perform  compliance  testing  as  specified  in  the  VV&A  plan. 

-  Perform  integration  testing  as  specified  in  the  integration  test  plan. 

—  Perform  risk  reduction  missions. 

-  Perform  validation  as  specified  in  the  VV&A  plan. 

STEP  5:  Execute  Distributed  Test  and  Analyze  Results 

Activity  5.1:  Execute  Distributed  Test 

-  Activity  Purpose:  exercise  all  distributed  test  participants  as  an  integrated  whole  to 
generate  required  outputs  and  achieve  the  stated  test  objectives. 

-  Activity  Inputs:  refined  test  procedures  and  tested  ADS  architecture  from  integration 
testing. 

-  Activity  Output:  raw  test  results.  The  following  considerations  apply  during 
execution: 

--  Pre-  and  post-test  briefings  are  essential. 

~  Centralized  test  control/management  and  execution  monitoring  must  be 
maintained  following  detailed  test  control  procedures  and  checklists. 

—  Data  collected  during  execution  are  used  to  support  both  quick-look  and  detailed 
post-test  analyses. 

-  Strict  attention  must  be  given  to  maintaining  the  security  posture  of  the  ADS 
architecture  during  execution. 

—  Conduct  an  after-action  review  immediately  after  the  test  in  order  to  gather 
important  information  from  each  facility,  to  formulate  a  course  of  action  for 
correcting  any  problems,  and  to  prepare  for  the  next  period  of  testing. 

Activity  5.2:  Process  Output 

-  Activity  Purpose:  post-process  (as  necessary)  the  output  collected  during  test 
execution. 

Activity  Input:  raw  test  results  from  test  execution. 

Activity  Output:  derived  test  results. 

Activity  5.3:  Prepare  Results 

-  Activity  Purpose:  (1)  evaluate  the  data  analysis  results  in  order  to  determine  if  all  test 
objectives  have  been  met  and  (2)  identify  legacy  products  and  make  them  available  to 
other  programs. 

-  Activity  Input:  derived  test  results  along  with  the  test  evaluation  criteria  from 
Activity  2.3. 

-  Activity  Output:  documented  test  results  and  legacy  products. 
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NOTE:  These  papers  and  reports  can  be  downloaded  from  the  JADS  Web  site: 
http://www.jads.abq.com/® 
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Appendix  E  -  ADS-Related  Web  Sites 


Air  Force  Agency  for  Modeling  and  Simulation  (AFAMS) 
http://www.afams.af.mil/ 

Air  Force  Modeling  and  Simulation  Resource  Repository  (AFMSRR) 
http://afmsrr.afams.af.mil/ 

Defense  Modeling  and  Simulation  Office  (DMSO) 
http://www.dmso.mil/ 

Defense  Modeling,  Simulation  and  Tactical  Technology  Information  Analysis  Center 

(DMSTTIAC) 

http://dmsttiac.hq.iitri.com/ 

Distributed  Simulation  Technology,  Inc.  (DiSTi),  HLA  training 
http://www.simuIation.com/ 

Electronic  Proving  Ground  (EPG) 
http://www.epg.army.mil/epg/ 

Extended  Air  Defense  Simulation  (EADSIM) 
http://www.smdc.army.mil/eadsim.html 

Extended  Air  Defense  Testbed  (EADTB) 
http://www.smdc.anriy.mil/eadtba~2.html 

High  Level  Architecture  (HLA) 
http://hla.dmso.mil/ 

Joint  Accreditation  Support  Activity  (JASA) 
http://www.nawcwpns.navy.mil/~jasa/ 

Joint  Advanced  Distributed  Simulation  (JADS) 
http://www.jads.abq.com/ 

Joint  Electronic  Combat  Test  Using  Simulation  (JECSIM) 
http://www.nawcwpns.navy.mil/~jecsim/ 

Joint  Modeling  and  Simulation  System  (JMASS) 
http://www.jmass.wpafb.af.mil/ 

Joint  National  Test  Facility  (JNTF) 
http://www.jntf.osd.mil/ 
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Joint  Simulation  System  (JSIMS) 
http://www.jsims.mil/ 

MAK  Technologies,  Inc.,  HLA  software 
http://www.mak.com/ 

National  Simulation  Center  (NSC) 
http://leav-www.army.mil/nsc/ 

Naval  Air  Warfare  Center  Weapon  Division,  modeling  and  simulation/interoperability 
http://sim.mugu.navy.mil/ 

Navy  Test  and  Evaluation  Modeling  and  Simulation  (TEMS) 
http://www.nawcad.navy.mil/tems/ 

Navy  Test  and  Evaluation  Repository  for  Models  and  Simulations  (NTERMS) 
http://nterms.mugu.navy.mil/ 

NPSNET  Research  Group,  Naval  Postgraduate  School 
http://www-npsnet.cs.nps.navy.mil/npsnet/ 

OriginalSim,  Inc.,  HLA  software  development 
http://www.originalsim.com/ 

Simulation  Interoperability  Standards  Organization  (SISO) 
http://siso.sc.ist.ucf.edu/ 

Simulation  Interoperability  Workshop  (SIW) 
http://siso.sc.ist.ucf.edu/siw/ 

Space  and  Missile  Defense  Battle  Lab  (SMDBL),  U.S.  Army  Space  and  Missile  Defense 

Command  (USASMDC) 

http://www.smdc.army.mil/SMDBL.HTML 

Synthetic  Scene  Generation  Model  (SSGM) 
http  ://vader.nrl  .navy.mil/ 

Theater  Air  Command  and  Control  Simulation  Facility  (TACCSF) 
http://www.taccsf.kirtland.af.mil/ 

Virtual  Proving  Ground  (VPG) 
http://vpg.tecom.army.mil/ 


